2024-09-17T17:46:52.037 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T17:46:52.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T17:46:52.143 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909201 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: skanta@redhat.com first_in_suite: false job_id: '7909201' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITjrLNOdh6K+wDSoG+QfQxbx8qn3Q8s7dtwEXTsC0n5E2zRx3gRpcXaAStmbQYd6E23/zgR7BpX2ZT5FlIfwoQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T17:46:52.143 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T17:46:52.144 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T17:46:52.144 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T17:46:52.146 INFO:teuthology.task.internal:Checking packages... 2024-09-17T17:46:52.168 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T17:46:52.168 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T17:46:52.168 INFO:teuthology.packaging:ref: None 2024-09-17T17:46:52.168 INFO:teuthology.packaging:tag: None 2024-09-17T17:46:52.168 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T17:46:52.168 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T17:46:52.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T17:46:52.322 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T17:46:52.323 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T17:46:52.331 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T17:46:52.331 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T17:46:52.372 INFO:teuthology.task.internal:Saving configuration 2024-09-17T17:46:52.384 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T17:46:52.399 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T17:46:52.427 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909201', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 17:41:15.101458', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITjrLNOdh6K+wDSoG+QfQxbx8qn3Q8s7dtwEXTsC0n5E2zRx3gRpcXaAStmbQYd6E23/zgR7BpX2ZT5FlIfwoQ='} 2024-09-17T17:46:52.427 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T17:46:52.435 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-17T17:46:52.435 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T17:46:52.489 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2a71b47be0>, signals=[15]) 2024-09-17T17:46:52.489 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T17:46:52.498 INFO:teuthology.task.internal:Opening connections... 2024-09-17T17:46:52.498 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-09-17T17:46:52.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:46:52.576 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T17:46:52.586 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-09-17T17:46:52.661 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-09-17T17:46:52.662 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-09-17T17:46:52.719 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-09-17T17:46:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T17:46:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-09-17T17:46:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T17:46:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T17:46:52.720 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T17:46:52.720 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-09-17T17:46:52.741 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T17:46:52.752 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T17:46:52.775 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T17:46:52.775 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-09-17T17:46:52.792 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T17:46:52.799 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T17:46:52.800 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-09-17T17:46:52.954 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T17:46:52.987 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T17:46:53.006 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T17:46:53.006 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T17:46:53.007 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T17:46:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:46:53.025 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-17T17:46:53.083 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-503.el9.x86_64 2024-09-17T17:46:53.083 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-503.el9.x86_64 2024-09-17T17:46:53.083 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-09-17T17:46:55.627 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-17T17:46:58.654 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-09-17T17:47:03.828 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-09-17T17:47:05.346 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 117 kB/s | 19 kB 00:00 2024-09-17T17:47:06.276 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-17T17:47:11.664 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T17:47:13.554 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T17:47:13.554 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T17:47:13.644 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:47:13.655 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:47:13.655 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-17T17:47:13.655 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-17T17:47:13.656 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:47:13.657 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 79 M 2024-09-17T17:47:13.657 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 126 M 2024-09-17T17:47:13.657 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:47:14.082 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 119 kB/s | 36 kB 00:00 2024-09-17T17:47:27.181 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.6 MB/s | 35 MB 00:13 2024-09-17T17:47:28.947 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 1.9 MB/s | 29 MB 00:14 2024-09-17T17:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:15 2024-09-17T17:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout:Total 5.1 MB/s | 79 MB 00:15 2024-09-17T17:47:29.783 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:47:30.154 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:47:30.154 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:47:31.098 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:47:31.099 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:47:33.528 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:47:34.220 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T17:47:34.324 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T17:47:37.150 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T17:47:37.286 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T17:47:42.112 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T17:47:42.145 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:47:46.984 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:48:25.282 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:48:25.322 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:48:25.661 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:48:25.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T17:48:25.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T17:48:25.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T17:48:26.506 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:48:26.507 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:48:26.861 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T17:48:26.952 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T17:48:26.953 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T17:48:26.953 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T17:48:26.953 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T17:48:26.953 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T17:48:26.953 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T17:48:26.953 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T17:48:26.954 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T17:48:27.013 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T17:48:27.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T17:48:27.013 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-09-17T17:48:27.045 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-09-17T17:48:27.045 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-09-17T17:48:27.045 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 82.el9 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8274670 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T17:48:27.046 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:48:27.047 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T17:48:27.048 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T17:48:27.048 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-508.el9.x86_64 2024-09-17T17:48:27.048 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T17:48:27.687 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-09-17T17:48:29.050 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T17:48:29.081 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-09-17T17:48:29.084 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-17T17:48:29.099 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.Ixw8pWlGgS 2024-09-17T17:48:29.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ixw8pWlGgS 2024-09-17T17:48:29.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.Ixw8pWlGgS 2024-09-17T17:48:29.312 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.Ixw8pWlGgS is 6KB 2024-09-17T17:48:29.362 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.Ixw8pWlGgS 2024-09-17T17:48:29.377 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T17:48:29.440 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T17:48:29.441 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T17:48:29.441 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T17:48:29.441 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T17:48:29.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T17:48:29.595 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-09-17T17:48:59.626 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T17:48:59.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:48:59.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:49:34.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-17T17:49:43.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:49:43.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:49:46.209 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-17T17:49:58.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:49:58.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:50:16.637 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-17T17:50:31.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:50:31.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:50:32.034 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-17T17:50:32.424 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:50:32.424 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T17:50:32.425 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-17T17:50:32.439 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-508.el9.x86_64 2024-09-17T17:50:32.440 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T17:50:32.440 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T17:50:32.440 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T17:50:33.440 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T17:50:33.441 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T17:50:33.535 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-09-17T17:50:33.566 DEBUG:teuthology.parallel:result is None 2024-09-17T17:50:33.566 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T17:50:33.575 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T17:50:33.575 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T17:50:33.590 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T17:50:33.619 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T17:50:33.667 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T17:50:33.668 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T17:50:33.699 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T17:50:33.717 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T17:50:33.717 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T17:50:33.765 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:50:33.775 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:50:33.777 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T17:50:33.785 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T17:50:33.785 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T17:50:33.847 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T17:50:33.863 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T17:50:33.863 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T17:50:33.907 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T17:50:33.994 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T17:50:34.044 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T17:50:34.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T17:50:34.173 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:50:34.174 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T17:50:34.242 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-09-17T17:50:34.345 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T17:50:34.704 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T17:50:34.711 INFO:teuthology.task.internal:Starting timer... 2024-09-17T17:50:34.712 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T17:50:34.763 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T17:50:34.811 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-09-17T17:50:34.864 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-09-17T17:50:34.866 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T17:50:34.866 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-09-17T17:50:34.885 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-09-17T17:50:34.885 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-09-17T17:50:34.885 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T17:50:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:50:34.954 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T17:50:34.954 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-09-17T17:50:35.039 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T17:50:35.049 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T17:50:35.061 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T17:50:35.062 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T17:54:20.973 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-09-17T17:54:20.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:54:20.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T17:54:21.068 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-17T17:54:21.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-17T17:54:21.104 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T17:54:21.116 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T17:54:21.116 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T17:54:21.117 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T17:54:21.176 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T17:54:21.190 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T17:54:21.214 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-09-17T17:54:21.225 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-09-17T17:54:21.238 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T17:54:21.252 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T17:54:21.314 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:54:21.317 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T17:54:21.318 INFO:teuthology.run_tasks:Running task install... 2024-09-17T17:54:21.326 DEBUG:teuthology.task.install:project ceph 2024-09-17T17:54:21.326 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T17:54:21.326 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T17:54:21.326 INFO:teuthology.task.install:Using flavor: default 2024-09-17T17:54:21.332 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T17:54:21.332 INFO:teuthology.task.install:extra packages: [] 2024-09-17T17:54:21.332 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T17:54:21.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T17:54:21.480 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T17:54:21.480 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T17:54:21.590 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T17:54:21.591 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:54:21.591 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T17:54:21.616 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T17:54:21.616 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T17:54:21.682 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T17:54:21.757 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T17:54:21.822 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-09-17T17:54:21.823 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-09-17T17:54:22.060 INFO:teuthology.orchestra.run.smithi157.stdout:45 files removed 2024-09-17T17:54:22.087 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-09-17T17:54:22.634 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 321 kB/s | 84 kB 00:00 2024-09-17T17:54:22.926 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-17T17:54:23.204 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T17:54:23.652 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.3 MB 00:00 2024-09-17T17:54:26.745 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-09-17T17:54:33.820 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.5 MB 00:03 2024-09-17T17:54:36.370 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-17T17:54:37.108 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 37 MB/s | 23 MB 00:00 2024-09-17T17:54:42.605 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-17T17:54:44.806 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:54:44.813 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:54:44.813 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:54:44.813 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:54:44.813 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:54:44.813 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T17:54:44.814 INFO:teuthology.orchestra.run.smithi157.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T17:54:44.815 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:54:44.816 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:54:44.817 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:54:44.817 INFO:teuthology.orchestra.run.smithi157.stdout:Install 31 Packages 2024-09-17T17:54:44.817 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-09-17T17:54:44.817 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:54:44.817 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 62 M 2024-09-17T17:54:44.818 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:54:45.639 INFO:teuthology.orchestra.run.smithi157.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-17T17:54:45.698 INFO:teuthology.orchestra.run.smithi157.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 432 kB/s | 25 kB 00:00 2024-09-17T17:54:45.790 INFO:teuthology.orchestra.run.smithi157.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 8.1 MB/s | 756 kB 00:00 2024-09-17T17:54:45.915 INFO:teuthology.orchestra.run.smithi157.stdout:(4/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 13 MB/s | 11 MB 00:00 2024-09-17T17:54:45.989 INFO:teuthology.orchestra.run.smithi157.stdout:(5/33): libradosstriper1-19.3.0-4975.g1889a165. 2.5 MB/s | 508 kB 00:00 2024-09-17T17:54:46.066 INFO:teuthology.orchestra.run.smithi157.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 591 kB/s | 45 kB 00:00 2024-09-17T17:54:46.141 INFO:teuthology.orchestra.run.smithi157.stdout:(7/33): python3-ceph-common-19.3.0-4975.g1889a1 1.9 MB/s | 143 kB 00:00 2024-09-17T17:54:46.366 INFO:teuthology.orchestra.run.smithi157.stdout:(8/33): ceph-common-19.3.0-4975.g1889a165.el9.x 17 MB/s | 22 MB 00:01 2024-09-17T17:54:46.391 INFO:teuthology.orchestra.run.smithi157.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 645 kB/s | 161 kB 00:00 2024-09-17T17:54:46.417 INFO:teuthology.orchestra.run.smithi157.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.3 MB/s | 321 kB 00:00 2024-09-17T17:54:46.492 INFO:teuthology.orchestra.run.smithi157.stdout:(11/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 9.5 MB/s | 5.5 MB 00:00 2024-09-17T17:54:46.517 INFO:teuthology.orchestra.run.smithi157.stdout:(12/33): python3-rbd-19.3.0-4975.g1889a165.el9. 2.4 MB/s | 305 kB 00:00 2024-09-17T17:54:46.542 INFO:teuthology.orchestra.run.smithi157.stdout:(13/33): python3-rgw-19.3.0-4975.g1889a165.el9. 792 kB/s | 99 kB 00:00 2024-09-17T17:54:46.634 INFO:teuthology.orchestra.run.smithi157.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 915 kB/s | 107 kB 00:00 2024-09-17T17:54:46.659 INFO:teuthology.orchestra.run.smithi157.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-17T17:54:46.685 INFO:teuthology.orchestra.run.smithi157.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2024-09-17T17:54:46.701 INFO:teuthology.orchestra.run.smithi157.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-09-17T17:54:46.743 INFO:teuthology.orchestra.run.smithi157.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-09-17T17:54:46.760 INFO:teuthology.orchestra.run.smithi157.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-09-17T17:54:46.785 INFO:teuthology.orchestra.run.smithi157.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-09-17T17:54:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:(21/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-17T17:54:46.828 INFO:teuthology.orchestra.run.smithi157.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 532 kB/s | 36 kB 00:00 2024-09-17T17:54:46.845 INFO:teuthology.orchestra.run.smithi157.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 803 kB/s | 47 kB 00:00 2024-09-17T17:54:46.895 INFO:teuthology.orchestra.run.smithi157.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.6 MB/s | 308 kB 00:00 2024-09-17T17:54:46.912 INFO:teuthology.orchestra.run.smithi157.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-17T17:54:46.929 INFO:teuthology.orchestra.run.smithi157.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 375 kB/s | 26 kB 00:00 2024-09-17T17:54:46.954 INFO:teuthology.orchestra.run.smithi157.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-17T17:54:46.988 INFO:teuthology.orchestra.run.smithi157.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T17:54:47.039 INFO:teuthology.orchestra.run.smithi157.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-17T17:54:47.089 INFO:teuthology.orchestra.run.smithi157.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T17:54:47.164 INFO:teuthology.orchestra.run.smithi157.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-17T17:54:47.289 INFO:teuthology.orchestra.run.smithi157.stdout:(32/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 16 MB/s | 3.3 MB 00:00 2024-09-17T17:54:47.347 INFO:teuthology.orchestra.run.smithi157.stdout:(33/33): librados2-19.3.0-4975.g1889a165.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-17T17:54:47.353 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:54:47.353 INFO:teuthology.orchestra.run.smithi157.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-17T17:54:47.467 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:54:47.503 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:54:47.503 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:54:48.210 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:54:48.211 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:54:49.753 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:54:49.840 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T17:54:49.869 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T17:54:49.903 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T17:54:49.925 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T17:54:49.951 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T17:54:50.127 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T17:54:50.180 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T17:54:50.546 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T17:54:50.637 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T17:54:50.730 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T17:54:50.820 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T17:54:50.847 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T17:54:50.912 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T17:54:50.947 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T17:54:51.001 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T17:54:51.079 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T17:54:51.108 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T17:54:51.173 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T17:54:51.207 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T17:54:51.325 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T17:54:51.357 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T17:54:51.398 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T17:54:51.431 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T17:54:51.468 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T17:54:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T17:54:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T17:54:51.877 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T17:54:51.923 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T17:54:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T17:54:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T17:54:52.089 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T17:54:52.592 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T17:54:52.617 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T17:54:52.689 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T17:54:52.759 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T17:54:52.794 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T17:54:55.070 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T17:54:55.107 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T17:54:55.728 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T17:54:55.769 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T17:54:55.861 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T17:54:55.861 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T17:54:55.861 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T17:54:55.861 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:54:55.895 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-09-17T17:55:05.892 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:06.920 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T17:55:06.945 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T17:55:06.946 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:06.946 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T17:55:06.946 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T17:55:06.946 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T17:55:06.946 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:06.948 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T17:55:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T17:55:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T17:55:08.166 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T17:55:08.167 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T17:55:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T17:55:08.169 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T17:55:08.170 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T17:55:09.104 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T17:55:09.104 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:09.104 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-09-17T17:55:09.104 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.104 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T17:55:09.105 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.106 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:09.107 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:09.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-09-17T17:55:10.040 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:10.172 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T17:55:10.173 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 46 M 2024-09-17T17:55:10.174 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 201 M 2024-09-17T17:55:10.175 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:10.469 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 665 kB/s | 162 kB 00:00 2024-09-17T17:55:10.503 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-09-17T17:55:10.620 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 423 kB/s | 64 kB 00:00 2024-09-17T17:55:11.703 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 31 MB/s | 46 MB 00:01 2024-09-17T17:55:11.705 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:11.706 INFO:teuthology.orchestra.run.smithi157.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-17T17:55:11.734 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:11.755 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:11.755 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:12.119 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:12.120 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:12.648 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:12.729 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T17:55:12.774 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T17:55:12.805 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T17:55:17.161 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T17:55:17.237 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T17:55:19.600 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T17:55:19.600 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T17:55:19.600 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T17:55:19.600 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:19.906 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:20.178 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-09-17T17:55:20.785 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:38 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:20.938 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:55:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T17:55:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T17:55:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:20.944 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout:Install 37 Packages 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 31 M 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 109 M 2024-09-17T17:55:20.945 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:21.529 INFO:teuthology.orchestra.run.smithi157.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 46 kB/s | 6.6 kB 00:00 2024-09-17T17:55:21.712 INFO:teuthology.orchestra.run.smithi157.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 4.7 MB/s | 1.5 MB 00:00 2024-09-17T17:55:21.762 INFO:teuthology.orchestra.run.smithi157.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-17T17:55:21.854 INFO:teuthology.orchestra.run.smithi157.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 3.1 MB/s | 291 kB 00:00 2024-09-17T17:55:21.921 INFO:teuthology.orchestra.run.smithi157.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-17T17:55:22.280 INFO:teuthology.orchestra.run.smithi157.stdout:(6/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 30 MB/s | 17 MB 00:00 2024-09-17T17:55:22.746 INFO:teuthology.orchestra.run.smithi157.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 284 kB/s | 253 kB 00:00 2024-09-17T17:55:22.972 INFO:teuthology.orchestra.run.smithi157.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 156 kB/s | 106 kB 00:00 2024-09-17T17:55:23.039 INFO:teuthology.orchestra.run.smithi157.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-17T17:55:23.456 INFO:teuthology.orchestra.run.smithi157.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 191 kB/s | 135 kB 00:00 2024-09-17T17:55:23.572 INFO:teuthology.orchestra.run.smithi157.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-17T17:55:23.615 INFO:teuthology.orchestra.run.smithi157.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-17T17:55:23.656 INFO:teuthology.orchestra.run.smithi157.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 835 kB/s | 35 kB 00:00 2024-09-17T17:55:23.682 INFO:teuthology.orchestra.run.smithi157.stdout:(14/37): python3-requests-2.25.1-8.el9.noarch.r 178 kB/s | 125 kB 00:00 2024-09-17T17:55:23.707 INFO:teuthology.orchestra.run.smithi157.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-17T17:55:23.740 INFO:teuthology.orchestra.run.smithi157.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-17T17:55:23.774 INFO:teuthology.orchestra.run.smithi157.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 553 kB/s | 51 kB 00:00 2024-09-17T17:55:23.799 INFO:teuthology.orchestra.run.smithi157.stdout:(18/37): python3-urllib3-1.26.5-6.el9.noarch.rp 282 kB/s | 215 kB 00:00 2024-09-17T17:55:23.824 INFO:teuthology.orchestra.run.smithi157.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-09-17T17:55:23.866 INFO:teuthology.orchestra.run.smithi157.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-17T17:55:23.884 INFO:teuthology.orchestra.run.smithi157.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 572 kB/s | 43 kB 00:00 2024-09-17T17:55:23.900 INFO:teuthology.orchestra.run.smithi157.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 647 kB/s | 11 kB 00:00 2024-09-17T17:55:23.929 INFO:teuthology.orchestra.run.smithi157.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 626 kB/s | 18 kB 00:00 2024-09-17T17:55:23.954 INFO:teuthology.orchestra.run.smithi157.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-17T17:55:23.980 INFO:teuthology.orchestra.run.smithi157.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-17T17:55:24.005 INFO:teuthology.orchestra.run.smithi157.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T17:55:24.030 INFO:teuthology.orchestra.run.smithi157.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-17T17:55:24.055 INFO:teuthology.orchestra.run.smithi157.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-09-17T17:55:24.089 INFO:teuthology.orchestra.run.smithi157.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-17T17:55:24.114 INFO:teuthology.orchestra.run.smithi157.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-09-17T17:55:24.140 INFO:teuthology.orchestra.run.smithi157.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-17T17:55:24.165 INFO:teuthology.orchestra.run.smithi157.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T17:55:24.190 INFO:teuthology.orchestra.run.smithi157.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T17:55:24.224 INFO:teuthology.orchestra.run.smithi157.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-17T17:55:24.249 INFO:teuthology.orchestra.run.smithi157.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-17T17:55:24.274 INFO:teuthology.orchestra.run.smithi157.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-17T17:55:24.299 INFO:teuthology.orchestra.run.smithi157.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-17T17:55:24.304 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:24.305 INFO:teuthology.orchestra.run.smithi157.stdout:Total 9.1 MB/s | 31 MB 00:03 2024-09-17T17:55:24.459 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:24.500 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:24.500 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:25.013 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:25.014 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:26.403 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:26.469 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T17:55:26.506 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T17:55:26.538 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T17:55:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T17:55:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T17:55:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T17:55:26.786 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T17:55:26.864 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T17:55:26.906 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T17:55:26.940 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T17:55:26.970 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T17:55:27.005 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T17:55:27.038 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T17:55:27.078 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T17:55:27.114 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T17:55:27.148 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T17:55:27.197 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T17:55:27.229 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T17:55:27.303 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T17:55:27.441 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T17:55:27.528 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T17:55:27.581 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T17:55:27.620 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T17:55:27.669 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T17:55:27.733 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T17:55:27.920 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T17:55:27.965 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T17:55:28.044 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T17:55:28.140 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T17:55:28.181 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T17:55:28.308 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T17:55:28.469 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T17:55:28.500 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T17:55:28.526 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T17:55:28.526 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:28.527 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T17:55:28.527 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T17:55:28.527 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T17:55:28.527 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:29.992 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T17:55:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T17:55:30.021 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:30.021 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T17:55:30.021 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T17:55:30.022 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T17:55:30.022 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:30.484 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T17:55:30.509 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:30.735 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T17:55:30.759 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:30.823 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T17:55:32.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T17:55:32.697 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T17:55:32.698 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T17:55:33.738 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T17:55:33.738 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:33.739 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T17:55:33.740 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T17:55:33.741 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T17:55:33.742 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T17:55:33.743 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T17:55:33.744 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T17:55:33.744 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:33.744 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:34.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-09-17T17:55:34.728 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:34.819 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:55:34.862 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:34.863 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:55:34.863 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:34.928 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-09-17T17:55:35.505 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:35.635 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:35.636 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:55:35.637 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:35.637 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 784 k 2024-09-17T17:55:35.637 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 791 k 2024-09-17T17:55:35.637 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:35.910 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-17T17:55:35.910 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:35.911 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-17T17:55:35.911 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:35.916 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:35.916 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:35.922 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:35.922 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:35.992 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:36.447 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:55:36.539 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:55:36.907 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:37.164 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:37.372 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T17:55:37.959 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:38.091 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:38.091 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:55:38.092 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:38.093 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 145 k 2024-09-17T17:55:38.093 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 432 k 2024-09-17T17:55:38.093 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:38.282 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 767 kB/s | 145 kB 00:00 2024-09-17T17:55:38.283 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:38.284 INFO:teuthology.orchestra.run.smithi157.stdout:Total 760 kB/s | 145 kB 00:00 2024-09-17T17:55:38.284 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:38.291 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:38.291 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:38.347 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:38.348 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:38.486 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:38.567 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T17:55:38.591 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T17:55:38.591 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:55:38.591 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T17:55:38.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:39.524 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T17:55:39.524 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:39.525 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:39.525 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:55:39.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:39.525 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:39.744 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-09-17T17:55:40.349 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:40.440 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:55:40.481 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:40.482 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:55:40.483 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:40.556 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T17:55:41.132 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:41.266 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:41.267 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:41.267 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:55:41.267 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:41.267 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T17:55:41.268 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-09-17T17:55:41.269 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:41.270 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 19 M 2024-09-17T17:55:41.270 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 159 M 2024-09-17T17:55:41.270 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:41.744 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4975.g188 111 kB/s | 16 kB 00:00 2024-09-17T17:55:41.769 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4975.g18 197 kB/s | 34 kB 00:00 2024-09-17T17:55:41.928 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-17T17:55:41.986 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-09-17T17:55:42.045 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T17:55:42.071 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-17T17:55:42.129 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-17T17:55:42.154 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-17T17:55:42.180 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T17:55:42.205 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-17T17:55:42.347 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 19 MB/s | 14 MB 00:00 2024-09-17T17:55:42.372 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.9 MB/s | 863 kB 00:00 2024-09-17T17:55:42.375 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:42.375 INFO:teuthology.orchestra.run.smithi157.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-17T17:55:42.423 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:42.442 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:42.442 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:42.742 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:42.743 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:43.349 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:43.414 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T17:55:43.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T17:55:43.495 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T17:55:43.535 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T17:55:43.640 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T17:55:43.778 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T17:55:43.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T17:55:43.939 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T17:55:43.968 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T17:55:44.000 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T17:55:47.173 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T17:55:47.221 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T17:55:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T17:55:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T17:55:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T17:55:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T17:55:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T17:55:49.427 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T17:55:49.427 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:49.427 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T17:55:49.428 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T17:55:49.429 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:49.429 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:55:49.674 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T17:55:50.245 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:08 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:55:50.377 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:55:50.378 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-17T17:55:50.378 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:55:50.378 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-17T17:55:50.378 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:55:50.378 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T17:55:50.379 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 38 M 2024-09-17T17:55:50.380 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 212 M 2024-09-17T17:55:50.381 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:55:50.911 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-17T17:55:50.970 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-17T17:55:50.995 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-17T17:55:51.188 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-09-17T17:55:51.279 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-17T17:55:51.305 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 59 kB/s | 18 kB 00:00 2024-09-17T17:55:51.347 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 242 kB/s | 38 kB 00:00 2024-09-17T17:55:51.439 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-17T17:55:51.539 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 250 kB 00:00 2024-09-17T17:55:51.622 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 5.1 MB 00:00 2024-09-17T17:55:53.048 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-17T17:55:53.114 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.7 MB/s | 4.9 MB 00:01 2024-09-17T17:55:53.115 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:55:53.116 INFO:teuthology.orchestra.run.smithi157.stdout:Total 14 MB/s | 38 MB 00:02 2024-09-17T17:55:53.472 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:55:53.498 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:55:53.498 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:55:53.925 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:55:53.925 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:55:54.687 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:55:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T17:55:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T17:55:54.904 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T17:55:54.939 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T17:55:55.123 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T17:55:55.154 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T17:55:55.231 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T17:55:55.879 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T17:55:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T17:55:57.284 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T17:55:58.324 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T17:55:58.354 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T17:56:00.859 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T17:56:01.238 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:01.239 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:01.617 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-09-17T17:56:02.219 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:20 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:02.365 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:56:02.366 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:02.367 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:02.368 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:02.368 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-09-17T17:56:02.368 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:02.368 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.2 M 2024-09-17T17:56:02.368 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-09-17T17:56:02.369 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:02.871 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 248 kB/s | 26 kB 00:00 2024-09-17T17:56:02.895 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 149 kB/s | 19 kB 00:00 2024-09-17T17:56:02.962 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 252 kB/s | 49 kB 00:00 2024-09-17T17:56:03.004 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-17T17:56:03.029 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-17T17:56:03.054 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T17:56:03.080 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 638 kB/s | 32 kB 00:00 2024-09-17T17:56:03.105 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 290 kB/s | 14 kB 00:00 2024-09-17T17:56:03.139 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-17T17:56:03.173 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-17T17:56:03.206 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 10 MB/s | 1.0 MB 00:00 2024-09-17T17:56:03.231 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.0 MB/s | 279 kB 00:00 2024-09-17T17:56:03.248 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-17T17:56:03.250 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:03.250 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-17T17:56:03.316 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:03.331 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:03.421 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:03.930 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:04.096 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T17:56:04.150 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T17:56:04.213 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T17:56:04.258 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T17:56:04.300 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T17:56:04.411 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T17:56:04.513 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T17:56:04.569 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T17:56:05.287 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T17:56:05.340 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T17:56:05.374 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T17:56:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T17:56:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T17:56:06.207 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T17:56:06.208 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T17:56:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T17:56:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:06.533 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:56:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:06.534 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:06.795 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T17:56:07.383 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:25 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:07.518 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T17:56:07.519 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:07.520 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-09-17T17:56:07.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:07.521 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 7.0 M 2024-09-17T17:56:07.521 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-09-17T17:56:07.521 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:07.849 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-17T17:56:07.899 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-17T17:56:07.925 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 692 kB/s | 154 kB 00:00 2024-09-17T17:56:07.950 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-17T17:56:07.992 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-17T17:56:09.309 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.7 MB/s | 6.0 MB 00:01 2024-09-17T17:56:09.311 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:09.312 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-09-17T17:56:09.384 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:09.395 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:09.395 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:09.497 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:09.497 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:09.975 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T17:56:10.019 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T17:56:10.547 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T17:56:10.639 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T17:56:10.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T17:56:10.731 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T17:56:11.499 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T17:56:11.499 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T17:56:11.499 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T17:56:11.499 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T17:56:11.500 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T17:56:11.500 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T17:56:11.774 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T17:56:11.774 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:11.774 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:11.774 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:11.775 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:12.044 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-09-17T17:56:12.628 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:12.761 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:12.761 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:12.761 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:56:12.761 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:12.761 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-09-17T17:56:12.762 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:12.763 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 979 k 2024-09-17T17:56:12.763 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.9 M 2024-09-17T17:56:12.763 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:13.019 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-17T17:56:13.444 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 1.5 MB/s | 899 kB 00:00 2024-09-17T17:56:13.445 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:13.445 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.4 MB/s | 979 kB 00:00 2024-09-17T17:56:13.458 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:13.481 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:13.481 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:13.564 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:13.564 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:13.751 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:13.889 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T17:56:13.969 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T17:56:14.648 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T17:56:14.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T17:56:14.923 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T17:56:14.923 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:14.923 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:14.923 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T17:56:14.923 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:14.924 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:15.212 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-09-17T17:56:15.811 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T17:56:15.950 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 285 k 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2024-09-17T17:56:15.951 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:16.181 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-17T17:56:16.182 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:16.182 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-17T17:56:16.182 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:16.190 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:16.190 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:16.223 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:16.224 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:16.443 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:56:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:56:16.524 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:56:16.524 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T17:56:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T17:56:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:17.516 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T17:56:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:17.517 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:17.786 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-09-17T17:56:18.381 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:18.518 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:18.519 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:18.520 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:18.520 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 126 k 2024-09-17T17:56:18.520 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 456 k 2024-09-17T17:56:18.520 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:18.736 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 587 kB/s | 126 kB 00:00 2024-09-17T17:56:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:Total 582 kB/s | 126 kB 00:00 2024-09-17T17:56:18.737 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:18.742 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:18.742 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:18.807 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:18.808 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:18.963 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:19.075 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:19.543 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:19.787 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:20.015 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-09-17T17:56:20.611 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:20.703 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:20.750 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:20.751 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:20.751 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:20.817 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-09-17T17:56:21.426 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:21.575 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:21.576 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:21.577 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:21.577 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 32 k 2024-09-17T17:56:21.577 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 142 k 2024-09-17T17:56:21.577 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:21.723 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 218 kB/s | 32 kB 00:00 2024-09-17T17:56:21.723 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:21.723 INFO:teuthology.orchestra.run.smithi157.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-17T17:56:21.724 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:21.727 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:21.728 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:21.743 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:21.743 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:21.830 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:21.938 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:22.300 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:22.546 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:22.547 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:22.547 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:22.547 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:56:22.547 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:22.547 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:22.720 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-09-17T17:56:23.297 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:23.386 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:23.428 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:23.429 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:23.429 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:23.494 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-09-17T17:56:24.068 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:24.158 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:24.201 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:24.202 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:24.203 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:24.265 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-09-17T17:56:24.843 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:24.934 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:24.978 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:24.979 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:24.979 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:25.051 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-09-17T17:56:25.636 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:25.727 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:25.772 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:25.773 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:25.773 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:25.857 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-09-17T17:56:26.434 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:26.567 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:26.567 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:26.568 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:26.634 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-09-17T17:56:27.220 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:27.310 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T17:56:27.354 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:27.355 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-17T17:56:27.355 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:27.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-09-17T17:56:28.008 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:28.143 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:28.143 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:28.143 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-17T17:56:28.143 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 85 k 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 231 k 2024-09-17T17:56:28.144 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:28.334 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 448 kB/s | 85 kB 00:00 2024-09-17T17:56:28.335 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:28.335 INFO:teuthology.orchestra.run.smithi157.stdout:Total 445 kB/s | 85 kB 00:00 2024-09-17T17:56:28.335 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:28.342 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:28.342 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:28.406 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:28.407 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:28.548 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:28.665 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:29.052 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:29.298 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:29.298 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:29.298 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:29.299 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:56:29.299 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:29.299 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:29.543 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-09-17T17:56:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:30.268 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:30.268 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:30.268 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.2 M 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 14 M 2024-09-17T17:56:30.269 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:30.588 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-17T17:56:30.588 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:30.589 INFO:teuthology.orchestra.run.smithi157.stdout:Total 10 MB/s | 3.2 MB 00:00 2024-09-17T17:56:30.589 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:30.598 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:30.598 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:30.679 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:30.679 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:31.118 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T17:56:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:32.338 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:32.338 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:32.338 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:32.338 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:56:32.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:32.339 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:32.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-09-17T17:56:33.198 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 05:54:42 PM UTC. 2024-09-17T17:56:33.331 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-17T17:56:33.332 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:33.333 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 170 k 2024-09-17T17:56:33.333 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 490 k 2024-09-17T17:56:33.333 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-17T17:56:33.541 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 818 kB/s | 170 kB 00:00 2024-09-17T17:56:33.542 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-17T17:56:33.542 INFO:teuthology.orchestra.run.smithi157.stdout:Total 813 kB/s | 170 kB 00:00 2024-09-17T17:56:33.542 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-17T17:56:33.549 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-17T17:56:33.549 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-17T17:56:33.613 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-17T17:56:33.613 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-17T17:56:33.768 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-17T17:56:33.880 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:34.285 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T17:56:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:34.566 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-17T17:56:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T17:56:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-17T17:56:34.566 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-17T17:56:34.821 DEBUG:teuthology.parallel:result is None 2024-09-17T17:56:34.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T17:56:34.960 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T17:56:34.982 INFO:teuthology.orchestra.run.smithi157.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T17:56:34.982 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T17:56:34.982 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T17:56:34.984 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T17:56:34.985 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:56:34.985 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T17:56:35.047 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T17:56:35.048 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:56:35.048 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T17:56:35.112 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T17:56:35.176 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T17:56:35.176 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:56:35.176 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T17:56:35.241 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T17:56:35.304 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T17:56:35.305 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:56:35.305 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T17:56:35.369 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T17:56:35.436 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T17:56:35.452 INFO:tasks.workunit:Pulling workunits from ref 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T17:56:35.452 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T17:56:35.452 INFO:tasks.workunit:timeout=3h 2024-09-17T17:56:35.452 INFO:tasks.workunit:cleanup=True 2024-09-17T17:56:35.453 DEBUG:teuthology.orchestra.run.smithi157:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T17:56:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T17:56:35.492 INFO:teuthology.orchestra.run.smithi157.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T17:56:35.492 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T17:56:35.547 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T17:56:35.547 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T17:56:35.603 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T17:56:35.659 INFO:tasks.workunit.client.0.smithi157.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T17:57:25.607 INFO:tasks.workunit.client.0.smithi157.stderr:Note: switching to '1889a165b5128787ec693450a3d1c5df5d6dc203'. 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.608 INFO:tasks.workunit.client.0.smithi157.stderr: git switch -c 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr:Or undo this operation with: 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr: git switch - 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-17T17:57:25.609 INFO:tasks.workunit.client.0.smithi157.stderr:HEAD is now at 1889a165b51 Merge branch 'wip-matanb-cleanup-snapmap' of https://github.com/Matan-B/ceph into wip-bharath10-testing-2024-09-17-1348 2024-09-17T17:57:25.611 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T17:57:25.668 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-17T17:57:25.668 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T17:57:25.725 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-17T17:57:25.726 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-17T17:57:25.726 DEBUG:teuthology.orchestra.run.smithi157:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-17T17:57:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T17:57:25.825 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/bad-inc-map 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:54181 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:54181 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-17T17:57:25.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-17T17:57:25.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none ' 2024-09-17T17:57:25.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:54181 ' 2024-09-17T17:57:25.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-17T17:57:25.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-17T17:57:25.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/bad-inc-map 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/bad-inc-map 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:57:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:57:25.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:57:25.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:57:25.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:57:25.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:57:25.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:57:25.850 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:57:25.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:57:25.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:57:25.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:57:25.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:57:25.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:57:25.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:57:25.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:57:25.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:57:25.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:57:25.854 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-17T17:57:25.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:57:25.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:25.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:25.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-17T17:57:25.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:57:25.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:57:25.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/bad-inc-map 2024-09-17T17:57:25.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:57:25.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:25.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:25.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27247 2024-09-17T17:57:25.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:57:25.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T17:57:25.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/bad-inc-map 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:57:25.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/bad-inc-map/a 2024-09-17T17:57:25.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-17T17:57:26.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:57:26.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:57:26.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T17:57:26.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:57:26.594 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-17T17:57:26.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:57:26.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get fsid 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:57:26.676 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:57:26.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get mon_host 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/bad-inc-map 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:57:26.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/bad-inc-map/x 2024-09-17T17:57:26.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:57:27.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:57:27.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:57:27.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-17T17:57:27.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-17T17:57:27.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/0 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 ' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:57:27.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:57:27.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/0 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ddb62a48-c8fb-403d-8f39-c6b8977670df 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ddb62a48-c8fb-403d-8f39-c6b8977670df' 2024-09-17T17:57:27.629 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 ddb62a48-c8fb-403d-8f39-c6b8977670df 2024-09-17T17:57:27.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:57:27.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAHw+lmPDRtJhAAbNpWvgKsqyG0W5jOfuPzTA== 2024-09-17T17:57:27.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAHw+lmPDRtJhAAbNpWvgKsqyG0W5jOfuPzTA=="}' 2024-09-17T17:57:27.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ddb62a48-c8fb-403d-8f39-c6b8977670df -i td/bad-inc-map/0/new.json 2024-09-17T17:57:27.926 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:57:27.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/0/new.json 2024-09-17T17:57:27.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHw+lmPDRtJhAAbNpWvgKsqyG0W5jOfuPzTA== --osd-uuid ddb62a48-c8fb-403d-8f39-c6b8977670df 2024-09-17T17:57:27.998 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:27.997+0000 7fd05e210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:28.003 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:28.002+0000 7fd05e210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:28.004 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:28.003+0000 7fd05e210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:28.005 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:28.004+0000 7fd05e210640 -1 bdev(0x556298bfdc00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:57:28.005 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:28.004+0000 7fd05e210640 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-17T17:57:33.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-17T17:57:33.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:57:33.816 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T17:57:33.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:57:33.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:57:34.338 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T17:57:34.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:57:34.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:57:34.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:57:34.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:57:34.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:57:34.363 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:34.361+0000 7f7bcd214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:34.365 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:34.363+0000 7f7bcd214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:34.365+0000 7f7bcd214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:57:34.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:34.770 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:57:34.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:57:34.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:34.771 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:35.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:35.722 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:35.721+0000 7f7bcd214640 -1 Falling back to public interface 2024-09-17T17:57:36.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:36.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:36.189 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T17:57:36.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:57:36.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:36.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:36.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:37.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:37.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:37.607 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T17:57:37.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:57:37.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:37.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:38.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:38.635 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:57:38.633+0000 7f7bcd214640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:57:39.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:39.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:39.032 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T17:57:39.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:57:39.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:39.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:39.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:40.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:40.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:40.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:57:40.663 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T17:57:40.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:40.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:41.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:42.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:42.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:42.061 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T17:57:42.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:57:42.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:42.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:42.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:43.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:43.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:43.468 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T17:57:43.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:57:43.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:43.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:43.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:44.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:44.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:44.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:57:44.862 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T17:57:44.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:44.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:45.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:46.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:46.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:46.348 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T17:57:46.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:57:46.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:46.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:46.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:47.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:47.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:47.760 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T17:57:47.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:57:47.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:47.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:48.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:49.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:49.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:49.185 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T17:57:49.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:57:49.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:49.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:49.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:50.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:50.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:50.922 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T17:57:50.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:57:50.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:50.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:51.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:52.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:52.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:52.461 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T17:57:52.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:57:52.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:52.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:52.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:53.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:53.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:53.876 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T17:57:53.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:57:53.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:53.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:54.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:55.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:55.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:55.506 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T17:57:55.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:57:55.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:55.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:55.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:56.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:56.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:56.895 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T17:57:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:57:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:56.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:57.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:57:58.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:57:58.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:57:58.385 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T17:57:58.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:57:58.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:57:58.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:57:59.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:00.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:00.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:00.630 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T17:58:00.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:58:00.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:00.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:58:01.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:02.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:02.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:02.778 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T17:58:02.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:58:02.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:02.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:58:03.219 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3296064385,v1:127.0.0.1:6803/3296064385] [v2:127.0.0.1:6804/3296064385,v1:127.0.0.1:6805/3296064385] exists,up ddb62a48-c8fb-403d-8f39-c6b8977670df 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-17T17:58:03.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/1 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 ' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-17T17:58:03.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-17T17:58:03.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/1 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c6656084-a06f-47a5-9055-1c6af3dd401c 2024-09-17T17:58:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c6656084-a06f-47a5-9055-1c6af3dd401c' 2024-09-17T17:58:03.227 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 c6656084-a06f-47a5-9055-1c6af3dd401c 2024-09-17T17:58:03.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:58:03.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQArw+lmt4ojDhAAPfFSJrldjBjrKGzbUfz63A== 2024-09-17T17:58:03.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQArw+lmt4ojDhAAPfFSJrldjBjrKGzbUfz63A=="}' 2024-09-17T17:58:03.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c6656084-a06f-47a5-9055-1c6af3dd401c -i td/bad-inc-map/1/new.json 2024-09-17T17:58:03.793 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T17:58:03.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/1/new.json 2024-09-17T17:58:03.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQArw+lmt4ojDhAAPfFSJrldjBjrKGzbUfz63A== --osd-uuid c6656084-a06f-47a5-9055-1c6af3dd401c 2024-09-17T17:58:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:03.829+0000 7fdd679c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:03.846 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:03.843+0000 7fdd679c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:03.847 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:03.844+0000 7fdd679c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:03.847 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:03.845+0000 7fdd679c5640 -1 bdev(0x558f5ae09c00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-17T17:58:03.847 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:03.845+0000 7fdd679c5640 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-17T17:58:09.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-17T17:58:09.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:58:09.551 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T17:58:09.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T17:58:09.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:58:10.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T17:58:10.083 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T17:58:10.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:58:10.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:58:10.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:58:10.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:58:10.108 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:10.106+0000 7ff0cfa1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:10.131 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:10.128+0000 7ff0cfa1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:10.132 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:10.129+0000 7ff0cfa1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:58:10.545 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:58:10.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:10.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:10.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:11.237 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:11.235+0000 7ff0cfa1b640 -1 Falling back to public interface 2024-09-17T17:58:11.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:11.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:11.955 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T17:58:11.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:58:11.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:11.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:12.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:13.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:13.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:13.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:58:13.365 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T17:58:13.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:13.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:13.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:14.318 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:14.316+0000 7ff0cfa1b640 -1 osd.1 0 log_to_monitors true 2024-09-17T17:58:14.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:14.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:14.763 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T17:58:14.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:58:14.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:14.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:15.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:16.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:16.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:16.289 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T17:58:16.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:58:16.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:16.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:16.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:17.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:17.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:17.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:58:17.688 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T17:58:17.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:17.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:18.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:19.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:19.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:19.109 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T17:58:19.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:58:19.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:19.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:19.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:20.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:20.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:20.543 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T17:58:20.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:58:20.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:20.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:20.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:21.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:21.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:21.941 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T17:58:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:58:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:21.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:22.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:23.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:23.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:23.401 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T17:58:23.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:58:23.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:23.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:23.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:24.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:24.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:24.921 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T17:58:24.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:58:24.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:24.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:25.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:26.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:26.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:26.508 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T17:58:26.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:58:26.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:26.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:27.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:28.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:28.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:28.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:58:28.030 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T17:58:28.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:28.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:29.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:30.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:30.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:30.236 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T17:58:30.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:58:30.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:30.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:31.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:32.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:32.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:32.013 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T17:58:32.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:58:32.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:32.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:32.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:33.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:33.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:33.955 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T17:58:33.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:58:33.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:33.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:34.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:35.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:35.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:35.787 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T17:58:35.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:58:35.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:35.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:36.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:37.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:37.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T17:58:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:58:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:58:38.158 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/959935303,v1:127.0.0.1:6811/959935303] [v2:127.0.0.1:6812/959935303,v1:127.0.0.1:6813/959935303] exists,up c6656084-a06f-47a5-9055-1c6af3dd401c 2024-09-17T17:58:38.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:58:38.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:58:38.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:58:38.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/2 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 ' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:58:38.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:58:38.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/2 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:58:38.165 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8273190e-1862-4455-801a-afee90f71f71 2024-09-17T17:58:38.166 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 8273190e-1862-4455-801a-afee90f71f71 2024-09-17T17:58:38.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8273190e-1862-4455-801a-afee90f71f71' 2024-09-17T17:58:38.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:58:38.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBOw+lmTD6LChAARuVgn3G/JGgEyrnug6gT+A== 2024-09-17T17:58:38.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBOw+lmTD6LChAARuVgn3G/JGgEyrnug6gT+A=="}' 2024-09-17T17:58:38.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8273190e-1862-4455-801a-afee90f71f71 -i td/bad-inc-map/2/new.json 2024-09-17T17:58:38.688 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T17:58:38.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/2/new.json 2024-09-17T17:58:38.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOw+lmTD6LChAARuVgn3G/JGgEyrnug6gT+A== --osd-uuid 8273190e-1862-4455-801a-afee90f71f71 2024-09-17T17:58:38.728 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:38.725+0000 7fbb913a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:38.767 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:38.764+0000 7fbb913a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:38.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:38.766+0000 7fbb913a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:38.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:38.766+0000 7fbb913a8640 -1 bdev(0x562033eabc00 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-17T17:58:38.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:38.766+0000 7fbb913a8640 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-17T17:58:44.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-17T17:58:44.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:58:44.755 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T17:58:44.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T17:58:44.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:58:45.299 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T17:58:45.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T17:58:45.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=69840f8f-a0d3-4bb6-9bc1-73277163f7db --auth-supported=none --mon-host=127.0.0.1:54181 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:58:45.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:58:45.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:58:45.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:58:45.323 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:45.321+0000 7f444500c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:45.337 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:45.335+0000 7f444500c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:45.338 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:45.336+0000 7f444500c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:58:45.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T17:58:45.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:58:45.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:58:45.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:58:45.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:58:45.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:45.769 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:58:45.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:58:45.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:45.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:46.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:46.681 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:46.679+0000 7f444500c640 -1 Falling back to public interface 2024-09-17T17:58:47.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:47.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:47.187 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T17:58:47.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:58:47.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:47.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:47.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:48.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:48.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:48.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:58:48.580 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T17:58:48.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:48.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:49.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:49.667 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:58:49.665+0000 7f444500c640 -1 osd.2 0 log_to_monitors true 2024-09-17T17:58:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:50.016 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T17:58:50.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:58:50.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:50.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:50.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:51.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:51.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:51.512 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T17:58:51.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:58:51.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:51.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:51.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:52.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:52.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T17:58:52.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:58:52.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:52.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:53.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:54.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:54.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:54.608 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T17:58:54.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:58:54.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:54.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:55.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:56.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:56.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:56.014 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T17:58:56.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:58:56.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:56.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:56.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:57.472 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T17:58:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:58:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:57.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:58:58.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:58:58.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:58:58.991 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T17:58:58.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:58:58.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:58:58.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:58:59.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:00.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:00.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:00.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:59:00.568 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T17:59:00.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:00.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:02.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:02.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:02.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:59:02.039 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T17:59:02.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:02.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:02.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:03.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:03.474 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T17:59:03.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:59:03.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:03.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:03.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:04.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:04.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:04.952 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T17:59:04.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:59:04.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:04.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:05.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:06.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:06.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:06.658 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T17:59:06.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T17:59:06.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:06.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:07.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:08.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:08.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:08.377 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T17:59:08.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T17:59:08.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:08.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:09.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:10.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:10.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:10.146 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T17:59:10.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T17:59:10.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:10.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:11.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:12.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:12.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:12.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T17:59:12.302 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T17:59:12.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:12.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:13.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:13.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:13.846 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T17:59:13.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T17:59:13.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:13.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:14.237 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3817112610,v1:127.0.0.1:6819/3817112610] [v2:127.0.0.1:6820/3817112610,v1:127.0.0.1:6821/3817112610] exists,up 8273190e-1862-4455-801a-afee90f71f71 2024-09-17T17:59:14.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:59:14.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:59:14.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:59:14.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-17T17:59:14.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-17T17:59:14.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 8 2024-09-17T17:59:15.255 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-17T17:59:15.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T17:59:16.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-17T17:59:17.790 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 1 2024-09-17T17:59:17.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-17T17:59:19.219 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 2 2024-09-17T17:59:19.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:59:24.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-17T17:59:24.234 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:59:24.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:59:24.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:24.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:24.655 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3296064385,v1:127.0.0.1:6803/3296064385] [v2:127.0.0.1:6804/3296064385,v1:127.0.0.1:6805/3296064385] exists,up ddb62a48-c8fb-403d-8f39-c6b8977670df 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:59:24.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-17T17:59:24.657 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:59:24.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:59:24.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:24.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T17:59:25.089 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/959935303,v1:127.0.0.1:6811/959935303] [v2:127.0.0.1:6812/959935303,v1:127.0.0.1:6813/959935303] exists,up c6656084-a06f-47a5-9055-1c6af3dd401c 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:59:25.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:59:25.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-17T17:59:25.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:59:25.092 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:59:25.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:25.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T17:59:25.515 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3817112610,v1:127.0.0.1:6819/3817112610] [v2:127.0.0.1:6820/3817112610,v1:127.0.0.1:6821/3817112610] exists,up 8273190e-1862-4455-801a-afee90f71f71 2024-09-17T17:59:25.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T17:59:25.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T17:59:25.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T17:59:25.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:14.822+0000 7f4433917640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.096+0000 7f443591b640 -1 osd.2 16 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.450+0000 7f443591b640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.968+0000 7f443591b640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:16.977+0000 7f443591b640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.195+0000 7f4433917640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.311+0000 7f4433917640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.518 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.637+0000 7f4433917640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.735+0000 7f4433917640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.840+0000 7f443591b640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.510+0000 7f443591b640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.963+0000 7f443591b640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.964+0000 7f443591b640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:19.522+0000 7f4433917640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-17T17:59:25.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-17T17:59:25.520 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:14.822+0000 7f4433917640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-17T17:59:25.520 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.096+0000 7f443591b640 10 osd.2 16 handle_osd_map bailing because last < start (16<17) 2024-09-17T17:59:25.520 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.450+0000 7f443591b640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:15.968+0000 7f443591b640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:16.977+0000 7f443591b640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.195+0000 7f4433917640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.311+0000 7f4433917640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.638+0000 7f4433917640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.735+0000 7f4433917640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:17.841+0000 7f443591b640 10 osd.2 22 handle_osd_map bailing because last < start (23<24) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.511+0000 7f443591b640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.964+0000 7f443591b640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:18.965+0000 7f443591b640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T17:59:19.522+0000 7f4433917640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-17T17:59:25.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-17T17:59:25.522 INFO:tasks.workunit.client.0.smithi157.stdout:success 2024-09-17T17:59:25.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-17T17:59:25.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=foo 2024-09-17T17:59:25.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-17T17:59:26.130 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' does not exist 2024-09-17T17:59:26.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-17T17:59:26.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:59:26.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:59:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:59:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:59:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:59:31.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:59:31.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-17T17:59:31.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-17T17:59:31.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:59:31.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-17T17:59:31.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:59:31.457 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:59:31.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:59:31.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:59:31.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:59:31.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:59:31.462 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:59:31.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:59:31.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:59:31.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:59:31.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:59:31.464 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:59:31.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:59:31.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:59:31.466 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:59:31.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:59:31.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-17T17:59:31.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:59:31.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:31.483 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:59:31.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-17T17:59:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:59:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:59:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T17:59:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/bad-inc-map 0 2024-09-17T17:59:31.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:59:31.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:59:31.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:59:31.487 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:59:31.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:59:31.488 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:59:31.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:59:31.489 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:59:31.489 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:59:31.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:59:31.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:59:31.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:59:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T17:59:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-17T17:59:31.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:59:31.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:31.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-17T17:59:31.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-17T17:59:31.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:59:31.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:59:31.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T17:59:31.497 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T17:59:31.497 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T17:59:31.565 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-17T17:59:31.565 DEBUG:teuthology.orchestra.run.smithi157:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-17T17:59:31.630 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T17:59:31.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/divergent-priors 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-17T17:59:31.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-17T17:59:31.639 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-17T17:59:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-17T17:59:31.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-17T17:59:31.640 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-17T17:59:31.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none ' 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-17T17:59:31.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_divergent_2 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_divergent_3 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_divergent_ec' 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-17T17:59:31.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T17:59:31.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T17:59:31.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T17:59:31.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T17:59:31.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T17:59:31.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T17:59:31.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T17:59:31.650 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T17:59:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T17:59:31.650 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T17:59:31.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T17:59:31.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T17:59:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T17:59:31.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T17:59:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T17:59:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:31.653 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:31.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T17:59:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T17:59:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T17:59:31.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-17T17:59:31.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T17:59:31.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:31.655 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:31.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33174 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-17T17:59:31.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-17T17:59:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-17T17:59:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-17T17:59:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-17T17:59:31.659 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-17T17:59:31.659 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-17T17:59:31.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-17T17:59:31.672 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-17T17:59:31.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T17:59:32.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T17:59:32.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:32.278 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T17:59:32.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:59:32.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get fsid 2024-09-17T17:59:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T17:59:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T17:59:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T17:59:32.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T17:59:32.362 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get mon_host 2024-09-17T17:59:32.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-17T17:59:32.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-17T17:59:32.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T17:59:32.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T17:59:32.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T17:59:32.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-17T17:59:32.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T17:59:33.308 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T17:59:33.308 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T17:59:33.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:59:33.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T17:59:33.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T17:59:33.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T17:59:33.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T17:59:33.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T17:59:33.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-17T17:59:33.348 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T17:59:33.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T17:59:33.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 148aef9a-f4de-4a72-bc44-cf0df81e99e4' 2024-09-17T17:59:33.349 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T17:59:33.349 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T17:59:33.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFw+lmCn/yFRAAD8soKT3d/Anwcp/8LvlpZA== 2024-09-17T17:59:33.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFw+lmCn/yFRAAD8soKT3d/Anwcp/8LvlpZA=="}' 2024-09-17T17:59:33.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 148aef9a-f4de-4a72-bc44-cf0df81e99e4 -i td/divergent-priors/0/new.json 2024-09-17T17:59:33.626 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:59:33.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-17T17:59:33.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFw+lmCn/yFRAAD8soKT3d/Anwcp/8LvlpZA== --osd-uuid 148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T17:59:33.674 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:33.672+0000 7f7a6281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:33.678 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:33.677+0000 7f7a6281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:33.679 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:33.678+0000 7f7a6281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:33.679 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:33.678+0000 7f7a6281f640 -1 bdev(0x55ea61299c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-17T17:59:33.679 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:33.678+0000 7f7a6281f640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-17T17:59:39.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-17T17:59:39.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T17:59:39.142 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T17:59:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T17:59:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T17:59:39.721 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T17:59:39.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T17:59:39.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T17:59:39.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T17:59:39.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T17:59:39.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T17:59:39.756 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:39.753+0000 7f3aaa20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:39.757 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:39.755+0000 7f3aaa20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:39.758 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:39.756+0000 7f3aaa20e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:40.148 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T17:59:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T17:59:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:40.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:40.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:41.094 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:41.092+0000 7f3aaa20e640 -1 Falling back to public interface 2024-09-17T17:59:41.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:41.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:41.539 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T17:59:41.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T17:59:41.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:41.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:41.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:42.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:42.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:42.977 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T17:59:42.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T17:59:42.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:42.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:43.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:43.746 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T17:59:43.743+0000 7f3aaa20e640 -1 osd.0 0 log_to_monitors true 2024-09-17T17:59:44.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:44.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:44.394 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T17:59:44.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T17:59:44.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:44.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:44.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:45.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:45.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:45.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T17:59:45.797 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T17:59:45.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:45.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:46.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:47.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:47.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:47.201 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T17:59:47.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T17:59:47.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:47.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:47.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:48.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:48.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:48.654 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T17:59:48.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T17:59:48.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:48.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:49.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:50.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:50.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:50.062 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T17:59:50.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T17:59:50.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:50.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:50.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:51.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:51.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:51.520 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T17:59:51.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T17:59:51.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:51.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:51.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:52.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:52.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:52.931 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T17:59:52.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T17:59:52.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:52.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:53.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:54.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:54.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:54.455 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T17:59:54.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T17:59:54.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:54.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:54.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:55.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:55.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:55.875 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T17:59:55.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T17:59:55.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:55.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:56.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:57.295 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T17:59:57.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T17:59:57.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:57.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T17:59:58.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T17:59:59.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T17:59:59.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T17:59:59.443 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T17:59:59.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T17:59:59.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T17:59:59.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:00:00.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:01.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:01.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:01.485 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:00:01.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:00:01.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:01.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:00:02.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:03.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:03.665 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:00:03.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:03.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:00:03.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:03.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:00:04.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:05.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:05.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:05.598 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:00:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:00:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:05.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:00:06.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:07.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:00:07.147 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:00:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:07.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:00:07.571 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/363915255,v1:127.0.0.1:6803/363915255] [v2:127.0.0.1:6804/363915255,v1:127.0.0.1:6805/363915255] exists,up 148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:00:07.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:00:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:00:07.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f6d829b3-29e5-4074-93d4-534092d988b4' 2024-09-17T18:00:07.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:00:07.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCnw+lmxbuSIxAA7wl8R+thEkd+vqRIe5ArpA== 2024-09-17T18:00:07.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCnw+lmxbuSIxAA7wl8R+thEkd+vqRIe5ArpA=="}' 2024-09-17T18:00:07.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6d829b3-29e5-4074-93d4-534092d988b4 -i td/divergent-priors/1/new.json 2024-09-17T18:00:08.054 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:00:08.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-17T18:00:08.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnw+lmxbuSIxAA7wl8R+thEkd+vqRIe5ArpA== --osd-uuid f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:00:08.103 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:08.101+0000 7fe33c287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:08.106 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:08.104+0000 7fe33c287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:08.107 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:08.105+0000 7fe33c287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:08.107 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:08.105+0000 7fe33c287640 -1 bdev(0x55dd38c72800 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:00:08.107 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:08.105+0000 7fe33c287640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-17T18:00:14.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-17T18:00:14.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:00:14.195 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:00:14.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:00:14.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:00:14.739 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:00:14.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:00:14.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:00:14.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:00:14.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:00:14.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:00:14.774 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:14.773+0000 7fb4fe083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:14.776 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:14.775+0000 7fb4fe083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:14.777 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:14.776+0000 7fb4fe083640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:00:15.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:15.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:00:15.190 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:00:15.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:15.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:16.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:16.115+0000 7fb4fe083640 -1 Falling back to public interface 2024-09-17T18:00:16.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:16.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:16.589 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:00:16.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:00:16.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:16.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:17.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:18.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:18.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:18.026 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:00:18.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:00:18.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:18.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:18.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:18.929 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:18.928+0000 7fb4fe083640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:00:19.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:19.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:19.437 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:00:19.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:00:19.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:19.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:20.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:21.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:21.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:21.126 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:00:21.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:00:21.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:21.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:21.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:22.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:22.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:22.599 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:00:22.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:00:22.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:22.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:23.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:24.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:24.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:24.029 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:00:24.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:00:24.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:24.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:24.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:25.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:25.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:25.461 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:00:25.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:00:25.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:25.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:25.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:00:26.966 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:00:26.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:26.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:27.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:28.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:28.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:28.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:00:28.383 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:00:28.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:28.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:28.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:29.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:29.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:29.886 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:00:29.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:00:29.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:29.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:30.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:31.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:31.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:31.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:00:31.309 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:00:31.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:31.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:31.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:32.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:32.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:32.896 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:00:32.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:00:32.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:32.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:33.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:34.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:34.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:34.746 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:00:34.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:00:34.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:34.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:35.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:36.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:36.411 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:00:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:00:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:36.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:37.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:38.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:38.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:38.205 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:00:38.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:00:38.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:38.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:39.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:40.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:40.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:40.187 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:00:40.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:00:40.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:40.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:40.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:41.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:41.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:41.667 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:00:41.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:00:41.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:41.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:00:42.087 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1829767729,v1:127.0.0.1:6811/1829767729] [v2:127.0.0.1:6812/1829767729,v1:127.0.0.1:6813/1829767729] exists,up f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:00:42.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:00:42.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:00:42.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:00:42.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:00:42.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:00:42.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:00:42.097 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:00:42.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 2b819a3c-0451-4f4d-b9ac-3a9f285840fa' 2024-09-17T18:00:42.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:00:42.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKw+lmtJy+BhAAh4x5MgmskqURVeFkguI6iw== 2024-09-17T18:00:42.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKw+lmtJy+BhAAh4x5MgmskqURVeFkguI6iw=="}' 2024-09-17T18:00:42.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2b819a3c-0451-4f4d-b9ac-3a9f285840fa -i td/divergent-priors/2/new.json 2024-09-17T18:00:42.611 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:00:42.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-17T18:00:42.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKw+lmtJy+BhAAh4x5MgmskqURVeFkguI6iw== --osd-uuid 2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:00:42.660 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:42.658+0000 7efcdbedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:42.662 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:42.659+0000 7efcdbedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:42.663 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:42.660+0000 7efcdbedc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:42.663 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:42.661+0000 7efcdbedc640 -1 bdev(0x56042df1bc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:00:42.663 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:42.661+0000 7efcdbedc640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-17T18:00:49.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-17T18:00:49.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:00:49.127 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:00:49.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:00:49.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:00:49.717 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:00:49.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:00:49.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:00:49.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:00:49.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:00:49.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:00:49.752 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:49.750+0000 7fc5b51b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:49.761 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:49.758+0000 7fc5b51b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:49.761 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:49.759+0000 7fc5b51b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:00:50.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:00:50.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:00:50.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:00:50.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:00:50.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:00:50.159 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:00:50.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:50.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:00:50.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:50.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:50.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:51.117 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:51.115+0000 7fc5b51b0640 -1 Falling back to public interface 2024-09-17T18:00:51.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:51.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:51.585 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:00:51.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:00:51.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:51.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:52.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:53.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:53.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:53.011 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:00:53.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:00:53.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:53.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:53.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:54.064 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:00:54.062+0000 7fc5b51b0640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:00:54.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:54.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:54.410 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:00:54.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:00:54.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:54.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:54.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:55.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:55.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:55.877 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:00:55.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:00:55.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:55.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:56.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:57.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:57.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:57.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:00:57.450 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:00:57.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:57.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:57.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:00:58.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:00:58.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:00:58.902 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:00:58.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:00:58.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:00:58.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:00:59.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:00.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:00.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:00.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:01:00.321 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:01:00.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:00.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:00.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:01.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:01.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:01.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:01:01.723 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:01:01.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:01.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:02.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:03.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:03.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:03.211 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:01:03.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:01:03.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:03.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:03.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:04.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:04.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:04.619 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:01:04.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:01:04.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:04.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:05.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:06.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:06.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:06.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:01:06.032 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:01:06.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:06.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:06.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:07.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:07.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:07.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:01:07.626 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:01:07.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:07.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:08.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:09.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:09.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:09.604 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:01:09.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:01:09.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:09.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:10.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:11.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:11.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:11.415 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:01:11.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:01:11.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:11.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:11.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:12.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:12.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:12.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:01:12.859 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:01:12.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:12.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:13.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:14.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:14.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:14.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:01:14.520 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:01:14.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:14.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:15.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:16.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:16.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:16.257 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:01:16.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:01:16.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:16.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:17.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:01:18.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:01:18.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:01:18.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:01:18.193 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:01:18.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:01:18.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:01:18.641 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/648233663,v1:127.0.0.1:6819/648233663] [v2:127.0.0.1:6820/648233663,v1:127.0.0.1:6821/648233663] exists,up 2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:01:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:01:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:01:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:01:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-17T18:01:19.240 INFO:tasks.workunit.client.0.smithi157.stderr:noout is set 2024-09-17T18:01:19.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-17T18:01:19.886 INFO:tasks.workunit.client.0.smithi157.stderr:noin is set 2024-09-17T18:01:19.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-17T18:01:20.614 INFO:tasks.workunit.client.0.smithi157.stderr:nodown is set 2024-09-17T18:01:20.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-17T18:01:20.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-17T18:01:21.149 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-17T18:01:21.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:01:22.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-17T18:01:22.767 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 size to 3 2024-09-17T18:01:22.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-17T18:01:23.580 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 2 2024-09-17T18:01:23.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-17T18:01:23.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:01:23.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:01:24.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:01:24.171 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:01:24.171 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:01:24.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:01:24.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:24.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:01:24.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-17T18:01:24.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-17T18:01:24.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-17T18:01:24.293 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:24.293 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:01:24.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-17T18:01:24.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-17T18:01:24.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974' 2024-09-17T18:01:24.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:24.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:01:24.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T18:01:24.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T18:01:24.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672974 2-64424509451' 2024-09-17T18:01:24.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:24.534 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-17T18:01:24.534 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:24.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:01:24.535 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-17T18:01:24.535 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:24.536 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836501 2024-09-17T18:01:24.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-17T18:01:24.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-17T18:01:24.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:24.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-17T18:01:24.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:01:25.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:01:25.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:26.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-17T18:01:26.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:26.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-17T18:01:26.399 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:26.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:01:26.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-17T18:01:26.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:26.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-17T18:01:26.406 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672974 2024-09-17T18:01:26.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-17T18:01:26.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:01:26.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-17T18:01:26.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:26.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T18:01:26.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:26.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:01:26.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T18:01:26.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:26.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T18:01:26.837 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509451 2024-09-17T18:01:26.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T18:01:26.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:01:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-17T18:01:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-17T18:01:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:01:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:01:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:01:27.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:01:27.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:01:27.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:01:27.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:01:27.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:01:27.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:01:27.348 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:01:27.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:01:27.762 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:01:27.762 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:01:27.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:01:27.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:27.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:01:27.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-17T18:01:27.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-17T18:01:27.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-17T18:01:27.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:27.884 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:01:28.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T18:01:28.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T18:01:28.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-09-17T18:01:28.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:28.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:01:28.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-17T18:01:28.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-17T18:01:28.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509453' 2024-09-17T18:01:28.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:28.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-17T18:01:28.126 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:28.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:01:28.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-17T18:01:28.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:28.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-17T18:01:28.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-17T18:01:28.128 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836504 2024-09-17T18:01:28.129 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:28.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-17T18:01:28.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:01:29.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:01:29.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:29.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-17T18:01:29.921 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:29.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T18:01:29.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:29.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:01:29.922 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T18:01:29.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:29.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T18:01:29.924 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672977 2024-09-17T18:01:29.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T18:01:29.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:01:30.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-17T18:01:30.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:30.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-17T18:01:30.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:30.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:01:30.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-17T18:01:30.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:30.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-17T18:01:30.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-17T18:01:30.412 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509453 2024-09-17T18:01:30.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:01:30.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509453 2024-09-17T18:01:30.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:01:30.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:01:30.860 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:01:31.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:01:31.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:01:31.427 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:01:31.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:01:31.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:01:31.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:01:31.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:01:31.428 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:01:31.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:01:31.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:01:31.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:01:31.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:01:32.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:01:32.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:01:32.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:01:32.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-17T18:01:32.363 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-17T18:01:32.761 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:01:32.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-17T18:01:32.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-17T18:01:32.772 INFO:tasks.workunit.client.0.smithi157.stdout:primary and soon to be divergent is 1 2024-09-17T18:01:32.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-17T18:01:33.140 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:01:33.140 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:01:21.938749+0000 0'0 29:20 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:04:38.783517+0000 0 0 2024-09-17T18:01:33.140 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:01:33.141 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:01:33.141 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-17T18:01:33.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-17T18:01:33.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-17T18:01:33.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-17T18:01:33.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-17T18:01:33.157 INFO:tasks.workunit.client.0.smithi157.stdout:writing initial objects 2024-09-17T18:01:33.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-17T18:01:33.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-17T18:01:33.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-17T18:01:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-17T18:01:33.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-17T18:01:33.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-17T18:01:33.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-17T18:01:33.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:33.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-17T18:01:34.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-17T18:01:34.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-17T18:01:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-17T18:01:34.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-17T18:01:34.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-17T18:01:34.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-17T18:01:34.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:34.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-17T18:01:35.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-17T18:01:35.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-17T18:01:35.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-17T18:01:35.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-17T18:01:35.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.724 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-17T18:01:35.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:35.866 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-17T18:01:36.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-17T18:01:36.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-17T18:01:36.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-17T18:01:36.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-17T18:01:36.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-17T18:01:36.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-17T18:01:36.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:36.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-17T18:01:37.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-17T18:01:37.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-17T18:01:37.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-17T18:01:37.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.500 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-17T18:01:37.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-17T18:01:37.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-17T18:01:37.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:37.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-17T18:01:38.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-17T18:01:38.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-17T18:01:38.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-17T18:01:38.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-17T18:01:38.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-17T18:01:38.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-17T18:01:38.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:38.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-17T18:01:39.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-17T18:01:39.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-17T18:01:39.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-17T18:01:39.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-17T18:01:39.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-17T18:01:39.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:39.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-17T18:01:40.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-17T18:01:40.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-17T18:01:40.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-17T18:01:40.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-17T18:01:40.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-17T18:01:40.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-17T18:01:40.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:40.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-17T18:01:41.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-17T18:01:41.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-17T18:01:41.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-17T18:01:41.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-17T18:01:41.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-17T18:01:41.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:41.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-17T18:01:42.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-17T18:01:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-17T18:01:42.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-17T18:01:42.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-17T18:01:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-17T18:01:42.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-17T18:01:42.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:42.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-17T18:01:43.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-17T18:01:43.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-17T18:01:43.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-17T18:01:43.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-17T18:01:43.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-17T18:01:43.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-17T18:01:43.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:43.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-17T18:01:44.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-17T18:01:44.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-17T18:01:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-17T18:01:44.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-17T18:01:44.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-17T18:01:44.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-17T18:01:44.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:44.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-17T18:01:45.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-17T18:01:45.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-17T18:01:45.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-17T18:01:45.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-17T18:01:45.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-17T18:01:45.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:45.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-17T18:01:46.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-17T18:01:46.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-17T18:01:46.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-17T18:01:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-17T18:01:46.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-17T18:01:46.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-17T18:01:46.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:46.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-17T18:01:47.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-17T18:01:47.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-17T18:01:47.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-17T18:01:47.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-17T18:01:47.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-17T18:01:47.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-17T18:01:47.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-17T18:01:47.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-17T18:01:48.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:01:48.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-17T18:01:48.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:01:48.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:01:48.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:01:48.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:01:48.100 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:01:48.100 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:01:48.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:01:48.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:01:48.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:01:48.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:48.539 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:01:48.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-17T18:01:48.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-17T18:01:48.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-17T18:01:48.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:48.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:01:48.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-17T18:01:48.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-17T18:01:48.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983' 2024-09-17T18:01:48.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:01:48.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672983 2-64424509460' 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-17T18:01:48.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:48.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:01:48.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-17T18:01:48.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:48.906 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836510 2024-09-17T18:01:48.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-17T18:01:48.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-17T18:01:48.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:49.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-17T18:01:49.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:01:50.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:01:50.338 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:01:50.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-17T18:01:50.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:50.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-17T18:01:50.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:50.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:01:50.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-17T18:01:50.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:50.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-17T18:01:50.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-17T18:01:50.782 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672983 2024-09-17T18:01:50.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:01:51.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-17T18:01:51.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:01:51.214 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-17T18:01:51.214 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:01:51.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:01:51.215 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-17T18:01:51.215 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:01:51.216 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509460 2024-09-17T18:01:51.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-17T18:01:51.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-17T18:01:51.217 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:01:51.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-17T18:01:51.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:01:51.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:01:51.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:01:52.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:01:52.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:01:52.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:01:52.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:01:52.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:01:53.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:01:53.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:01:53.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:01:53.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-17T18:01:53.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-17T18:01:53.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-17T18:01:53.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-17T18:01:53.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:01:53.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-17T18:01:53.545 INFO:tasks.workunit.client.0.smithi157.stdout:blackholing osds 0 2 2024-09-17T18:01:53.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-17T18:01:53.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-17T18:01:53.917 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:01:53.917 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-17T18:01:21.938749+0000 29'100 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:04:38.783517+0000 0 0 2024-09-17T18:01:53.917 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:01:53.918 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:01:53.918 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:01:53.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-17T18:01:53.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-17T18:01:53.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:01:53.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-17T18:01:53.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-17T18:01:54.005 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:01:54.006 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:01:54.006 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:01:54.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:01:54.014 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:01:54.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:01:54.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-17T18:01:54.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-17T18:01:54.092 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:01:54.092 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:01:54.093 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:01:54.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-17T18:01:54.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-17T18:01:54.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-17T18:01:54.101 INFO:tasks.workunit.client.0.smithi157.stdout:writing divergent object 2024-09-17T18:01:54.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-17T18:01:54.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-17T18:01:54.102 INFO:tasks.workunit.client.0.smithi157.stdout:create missing divergent object 2024-09-17T18:01:54.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=test 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=existing_99 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/divergent-priors 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T18:01:54.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T18:01:54.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds test existing_99 2024-09-17T18:01:54.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=test 2024-09-17T18:01:54.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=existing_99 2024-09-17T18:01:54.105 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map test existing_99 2024-09-17T18:01:54.105 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T18:01:54.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T18:01:54.588 INFO:tasks.workunit.client.0.smithi157.stderr:0 2024-09-17T18:01:54.588 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:01:54.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0' '2') 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T18:01:54.589 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/divergent-priors/1/type 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:01:54.591 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:01:54.592 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:01:54.592 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.33174/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:01:54.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T18:01:54.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok injectdataerr test existing_99 2024-09-17T18:01:54.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-17T18:01:54.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-17T18:02:04.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-17T18:02:04.794 INFO:tasks.workunit.client.0.smithi157.stderr:rados: no process found 2024-09-17T18:02:04.794 INFO:tasks.workunit.client.0.smithi157.stdout:killing all the osds 2024-09-17T18:02:04.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-17T18:02:04.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-17T18:02:05.177 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:02:05.177 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2024-09-17T18:01:54.907439+0000 29'101 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:04:38.783517+0000 0 0 2024-09-17T18:02:05.178 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:02:05.178 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:02:05.178 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:02:05.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-17T18:02:05.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:02:05.189 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:02:05.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:02:05.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:02:05.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:02:05.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:02:05.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-17T18:02:05.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-17T18:02:05.857 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:02:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-17T18:02:05.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-17T18:02:06.375 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:02:06.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-17T18:02:06.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-17T18:02:06.893 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:02:06.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:06.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-17T18:02:07.435 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already out. 2024-09-17T18:02:07.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:07.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-17T18:02:08.004 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already out. 2024-09-17T18:02:08.016 INFO:tasks.workunit.client.0.smithi157.stdout:bringing up non_divergent 0 2 2024-09-17T18:02:08.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-17T18:02:08.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-17T18:02:08.387 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:02:08.387 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2024-09-17T18:01:54.907439+0000 29'101 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:04:38.783517+0000 0 0 2024-09-17T18:02:08.387 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:02:08.387 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:02:08.387 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:02:08.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:02:08.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:02:08.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:02:08.406 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:08.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:02:08.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:02:08.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:02:08.411 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:02:08.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:02:08.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:02:08.436 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:08.434+0000 7f90b47d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:08.446 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:08.444+0000 7f90b47d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:08.447 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:08.445+0000 7f90b47d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:08.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:02:08.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:02:08.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:02:08.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:02:08.886 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:02:08.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:02:08.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:08.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:02:08.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:08.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:02:09.312 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:09.545 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:09.543+0000 7f90b47d9640 -1 Falling back to public interface 2024-09-17T18:02:10.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:10.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:10.313 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:02:10.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:02:10.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:10.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:02:10.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:11.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:11.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:11.757 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:02:11.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:02:11.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:11.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:02:12.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:12.650 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:12.649+0000 7f90b47d9640 -1 osd.0 29 log_to_monitors true 2024-09-17T18:02:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:02:13.159 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:02:13.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:13.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:02:13.559 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2801159071,v1:127.0.0.1:6803/2801159071] [v2:127.0.0.1:6804/2801159071,v1:127.0.0.1:6805/2801159071] exists,up 148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:02:13.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:02:13.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:02:13.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:02:13.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:02:13.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:02:13.574 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:02:13.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:02:13.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:02:13.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-17T18:02:13.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:02:13.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:02:13.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:02:13.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:02:13.596 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:13.595+0000 7f715c219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:13.618 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:13.617+0000 7f715c219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:13.619 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:13.618+0000 7f715c219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:13.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:02:13.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:02:13.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:02:13.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:02:13.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:02:13.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:13.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:02:13.992 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:02:13.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:13.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:02:14.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:14.714 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:14.713+0000 7f715c219640 -1 Falling back to public interface 2024-09-17T18:02:15.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:15.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:15.426 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:02:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:02:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:15.427 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:02:15.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:16.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:16.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:16.887 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:02:16.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:02:16.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:16.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:02:17.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:02:18.263 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:18.262+0000 7f715c219640 -1 osd.2 29 log_to_monitors true 2024-09-17T18:02:18.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:02:18.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:02:18.329 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:02:18.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:02:18.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:02:18.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:02:18.760 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1501073209,v1:127.0.0.1:6811/1501073209] [v2:127.0.0.1:6812/1501073209,v1:127.0.0.1:6813/1501073209] exists,up 2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:02:18.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:02:18.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:02:18.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:02:18.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:18.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-17T18:02:19.288 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already in. 2024-09-17T18:02:19.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:19.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-17T18:02:19.978 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already in. 2024-09-17T18:02:19.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:02:19.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-17T18:02:19.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:02:19.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:02:19.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:02:19.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:02:19.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:02:19.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:02:19.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:02:19.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:02:19.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:02:20.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:02:20.508 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:02:20.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-17T18:02:20.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-17T18:02:20.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-17T18:02:20.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:02:20.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:02:20.717 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:02:20.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:02:20.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:02:20.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:02:20.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:02:20.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:02:20.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:02:20.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:02:20.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-17T18:02:20.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:02:20.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-17T18:02:20.842 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 154618822660 2024-09-17T18:02:20.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-17T18:02:20.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:02:21.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-17T18:02:21.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:02:22.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:02:22.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:02:22.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-17T18:02:22.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:02:22.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-17T18:02:22.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:02:22.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:02:22.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-17T18:02:22.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:02:22.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-17T18:02:22.715 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 163208757251 2024-09-17T18:02:22.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-17T18:02:22.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:02:23.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-17T18:02:23.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:02:23.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:23.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:23.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:23.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:24.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:24.056 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:24.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:24.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:24.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:24.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:02:24.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:02:24.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:02:24.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:24.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:02:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:24.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:24.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:24.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:24.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:24.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:24.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:25.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:25.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:25.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:25.075 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:25.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:25.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:26.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:26.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:26.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:02:26.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:26.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:26.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:26.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:26.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:26.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:26.758 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:27.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:27.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:27.287 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:27.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:27.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:27.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:02:27.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:27.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:02:28.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:28.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:28.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:28.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:28.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:28.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:29.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:29.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:02:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:29.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:02:30.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:30.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:30.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:30.922 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:30.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:30.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:31.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:31.449 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:31.449 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:31.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:31.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:31.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:02:31.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:31.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:02:33.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:33.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:33.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:33.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:33.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:33.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:33.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:33.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:33.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:33.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:33.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:33.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:34.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:34.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:34.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:35.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:35.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:35.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:02:35.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:35.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:02:38.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:38.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:38.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:38.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:38.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:38.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:38.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:39.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:39.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:39.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:39.212 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:39.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:39.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:39.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:02:39.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:39.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:02:46.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:46.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:46.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:46.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:46.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:46.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:46.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:46.162 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:46.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:46.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:46.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:46.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:47.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:47.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:47.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:47.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:47.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:02:47.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:02:47.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:02:55.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:02:55.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:02:55.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:02:55.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:02:55.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:02:55.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:02:55.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:02:55.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:02:55.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:02:55.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:02:56.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:02:56.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:02:56.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:02:56.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:02:57.006 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "88c00761-72e4-4067-a669-e88ff6e0b12b", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:02:57.004314+0000", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:02:57.007 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:02:57.008 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:02:57.009 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:02:57.010 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:02:57.011 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T17:59:31.696982Z", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T17:59:31.696982Z", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:02:57.012 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:02:57.013 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.014 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.015 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 42, 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T17:59:32.546616+0000", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:02:20.053632+0000", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:02:18.468490+0000", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:02:19.631779+0000", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:02:57.016 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 9, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:02:57.017 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:01:20.903054+0000", 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 1, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool", 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:02:57.018 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 1, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 1, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 1, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 1, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 1, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:02:57.019 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "29", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 0, 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 0, 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:02:57.020 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-17T18:02:57.021 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:02:57.022 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {}, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "read_balance": { 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "score_type": "Fair distribution", 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "score_acting": 2.9900000095367432, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "score_stable": 2.9900000095367432, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_score": 0.67000001668930054, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_acting": 2, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_stable": 2, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity_weighted": 1, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity": 1, 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity_weighted": 1 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:02:57.023 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "148aef9a-f4de-4a72-bc44-cf0df81e99e4", 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 36, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 40, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 30, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:02:57.024 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:02:57.025 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.026 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.027 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.028 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2801159071 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/2801159071", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/2801159071", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2801159071", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2801159071", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:02:57.029 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "f6d829b3-29e5-4074-93d4-534092d988b4", 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 23, 2024-09-17T18:02:57.030 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 31, 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.031 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.032 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.033 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1829767729 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.034 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/1829767729", 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/1829767729", 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1829767729", 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1829767729", 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "2b819a3c-0451-4f4d-b9ac-3a9f285840fa", 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:02:57.035 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 38, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 0, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 32, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.036 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:02:57.037 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.038 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:02:57.039 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1501073209 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.040 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/1501073209", 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/1501073209", 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1501073209", 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1501073209", 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:02:57.041 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:02:05.619932+0000", 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:00:05.770498+0000", 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:02:06.201933+0000", 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:02:57.042 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:00:41.404915+0000", 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.044 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:02:06.716540+0000", 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:01:17.156073+0000", 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:02:57.045 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:02:57.046 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:02:57.047 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/2801159071,v1:127.0.0.1:6805/2801159071]", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:02:57.048 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T17:59:38.513501Z", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:02:57.049 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/2801159071,v1:127.0.0.1:6803/2801159071]", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2801159071,v1:127.0.0.1:6809/2801159071]", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2801159071,v1:127.0.0.1:6807/2801159071]", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:02:57.050 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/1829767729,v1:127.0.0.1:6813/1829767729]", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:02:57.051 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:02:57.052 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:00:13.480091Z", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:02:57.053 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/1829767729,v1:127.0.0.1:6811/1829767729]", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1829767729,v1:127.0.0.1:6817/1829767729]", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1829767729,v1:127.0.0.1:6815/1829767729]", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:02:57.054 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/1501073209,v1:127.0.0.1:6813/1501073209]", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:02:57.055 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:02:57.056 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:00:48.489914Z", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/1501073209,v1:127.0.0.1:6811/1501073209]", 2024-09-17T18:02:57.057 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1501073209,v1:127.0.0.1:6817/1501073209]", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1501073209,v1:127.0.0.1:6815/1501073209]", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:02:57.058 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:02:57.059 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 36 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 39 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.060 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 42, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:02:57.061 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.062 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.063 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:02:57.064 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.065 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:02:57.066 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.067 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:02:57.068 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.069 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.070 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.071 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:02:57.072 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:02:57.073 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:02:57.074 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T17:59:32:522109+0000", 2024-09-17T18:02:57.075 INFO:tasks.workunit.client.0.smithi157.stderr:report 4265382112 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:02:57.076 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:02:57.077 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 1, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48300, 2024-09-17T18:02:57.078 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 100, 2024-09-17T18:02:57.079 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 100, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:02:57.080 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:02:57.081 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.082 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 16, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 16, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:02:57.083 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83688, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2128, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489112, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036850688, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2179072, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 425358, 2024-09-17T18:02:57.084 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 1, 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:02:57.085 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:02:57.086 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83688, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2128, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489112, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036850688, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:02:57.087 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2179072, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 425358, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 1, 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:02:57.088 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:02:57.089 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48300, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:02:57.090 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 100, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 100, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:02:57.091 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:02:57.092 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1228800, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 144900, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 16, 2024-09-17T18:02:57.093 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 16, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 154618822667 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:02:57.094 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672986 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 163208757258 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 1 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:02:57.095 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 1, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:02:57.096 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 4, 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:01:43.337928+0000", 2024-09-17T18:02:57.097 INFO:tasks.workunit.client.0.smithi157.stdout: "services": { 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": { 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "daemons": { 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": "", 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "2": { 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "start_epoch": 0, 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "start_stamp": "0.000000", 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "gid": 0, 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "metadata": {}, 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: "task_status": {} 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.098 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 91, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 377, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 519, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:02:57.099 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:02:57.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:02:57.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-17T18:02:57.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-17T18:02:57.100 INFO:tasks.workunit.client.0.smithi157.stdout:writing non-divergent object existing_4 2024-09-17T18:02:57.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-17T18:02:57.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-17T18:02:57.433 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:02:57.433 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:02:57.433 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:02:57.433 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:02:57.433 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:02:57.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-17T18:02:57.578 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery 2024-09-17T18:02:57.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-17T18:02:57.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-17T18:02:57.944 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:02:57.944 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:02:57.944 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:02:57.945 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:02:57.945 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:02:57.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:57.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-17T18:02:57.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:02:57.956 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-17T18:02:57.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok set_recovery_delay 100000 2024-09-17T18:02:58.036 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:02:58.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-17T18:02:58.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok set_recovery_delay 100000 2024-09-17T18:02:58.124 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:02:58.130 INFO:tasks.workunit.client.0.smithi157.stdout:revive divergent 1 2024-09-17T18:02:58.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-17T18:02:58.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-17T18:02:58.507 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:02:58.507 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 29'100 42:130 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:02:58.507 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:02:58.507 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:02:58.508 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:02:58.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-17T18:02:59.153 INFO:tasks.workunit.client.0.smithi157.stderr:noup is set 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:02:59.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:02:59.169 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:02:59.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:02:59.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:02:59.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:02:59.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:02:59.171 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:02:59.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:02:59.203 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:59.200+0000 7fe43520b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:59.212 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:59.210+0000 7fe43520b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:59.213 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:02:59.211+0000 7fe43520b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:02:59.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-17T18:02:59.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-17T18:03:00.314 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:00.312+0000 7fe43520b640 -1 Falling back to public interface 2024-09-17T18:03:03.259 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:03.257+0000 7fe43520b640 -1 osd.1 29 log_to_monitors true 2024-09-17T18:03:04.577 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery divergent 2024-09-17T18:03:04.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-17T18:03:04.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:04.965 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:04.965 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:03:04.965 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:04.965 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:04.965 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:04.977 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:03:04.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-17T18:03:04.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok set_recovery_delay 100000 2024-09-17T18:03:05.058 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:03:05.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-17T18:03:05.849 INFO:tasks.workunit.client.0.smithi157.stderr:noup is unset 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:05.860 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:05.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:03:05.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:05.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:06.270 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2801159071,v1:127.0.0.1:6803/2801159071] [v2:127.0.0.1:6804/2801159071,v1:127.0.0.1:6805/2801159071] exists,up 148aef9a-f4de-4a72-bc44-cf0df81e99e4 2024-09-17T18:03:06.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:03:06.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:06.272 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:06.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:03:06.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:06.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:03:06.742 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2864524863,v1:127.0.0.1:6819/2864524863] [v2:127.0.0.1:6820/2864524863,v1:127.0.0.1:6821/2864524863] exists,up f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:03:06.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:03:06.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:03:06.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:03:06.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:06.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:03:06.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:06.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:03:07.202 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/1501073209,v1:127.0.0.1:6811/1501073209] [v2:127.0.0.1:6812/1501073209,v1:127.0.0.1:6813/1501073209] exists,up 2b819a3c-0451-4f4d-b9ac-3a9f285840fa 2024-09-17T18:03:07.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:03:07.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:03:07.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:03:07.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:07.575 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:07.575 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:03:07.575 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:07.575 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:07.576 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:07.586 INFO:tasks.workunit.client.0.smithi157.stdout:wait for peering 2024-09-17T18:03:07.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-17T18:03:07.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:07.956 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:07.956 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:03:07.957 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:07.957 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:07.957 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:07.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-17T18:03:08.076 INFO:tasks.workunit.client.0.smithi157.stdout:killing divergent 1 2024-09-17T18:03:08.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-17T18:03:08.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:08.449 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:08.449 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:03:08.449 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:08.449 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:08.450 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:08.460 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-17T18:03:08.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:03:08.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:03:08.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:03:08.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:03:08.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:03:08.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:03:08.566 INFO:tasks.workunit.client.0.smithi157.stdout:reviving divergent 1 2024-09-17T18:03:08.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-17T18:03:08.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:08.938 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:08.938 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:02:20.411407+0000 42'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:46:40.218441+0000 0 0 2024-09-17T18:03:08.938 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:08.938 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:08.938 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:08.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:03:08.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:03:08.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:03:08.954 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:03:08.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:03:08.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:03:08.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:03:08.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:03:08.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:03:08.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:03:08.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:03:08.986 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:08.985+0000 7fc4e8143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:08.988 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:08.987+0000 7fc4e8143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:08.989 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:08.988+0000 7fc4e8143640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:09.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:03:09.399 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:09.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:09.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:03:09.838 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2864524863,v1:127.0.0.1:6819/2864524863] [v2:127.0.0.1:6820/2864524863,v1:127.0.0.1:6821/2864524863] exists,up f6d829b3-29e5-4074-93d4-534092d988b4 2024-09-17T18:03:09.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:03:09.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:03:09.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:03:09.839 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-17T18:03:10.361 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:10.360+0000 7fc4e8143640 -1 Falling back to public interface 2024-09-17T18:03:13.016 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:13.014+0000 7fc4e8143640 -1 osd.1 48 log_to_monitors true 2024-09-17T18:03:29.840 INFO:tasks.workunit.client.0.smithi157.stdout:allowing recovery 2024-09-17T18:03:29.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-17T18:03:29.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:30.223 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:30.224 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-17T18:03:14.563746+0000 48'102 52:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:13:59.557362+0000 0 0 2024-09-17T18:03:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:30.224 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:30.224 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:30.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-17T18:03:30.235 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-17T18:03:30.348 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:03:30.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-17T18:03:30.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-17T18:03:30.478 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:03:30.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-17T18:03:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-17T18:03:30.598 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:03:30.607 INFO:tasks.workunit.client.0.smithi157.stdout:reading divergent objects 2024-09-17T18:03:30.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-17T18:03:30.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-17T18:03:30.979 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:03:30.979 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-17T18:03:14.563746+0000 48'102 52:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:01:20.992526+0000 0'0 2024-09-17T18:01:20.992526+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:13:59.557362+0000 0 0 2024-09-17T18:03:30.980 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:03:30.980 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:03:30.980 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:03:30.991 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-17T18:03:30.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-17T18:03:30.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:03:30.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-17T18:03:31.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:03:31.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-17T18:03:31.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:03:31.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-17T18:03:31.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:03:31.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-17T18:03:31.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-17T18:03:31.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-17T18:03:31.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-09-17T18:01:54.134280+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-09-17T18:01:54.134280+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-17T18:03:31.144 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:03:31.145 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-09-17T18:03:31.145 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4626.0:1 2024-09-17T18:01:54.134280+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:03:31.145 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:03:06.829+0000 7fe418bbc640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-17T18:03:31.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-17T18:03:31.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:03:31.150 INFO:tasks.workunit.client.0.smithi157.stdout:success 2024-09-17T18:03:31.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-17T18:03:31.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-17T18:03:31.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-17T18:03:31.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-17T18:03:31.948 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' does not exist 2024-09-17T18:03:31.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-17T18:03:31.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:03:31.960 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:03:31.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:03:31.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:03:31.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:03:37.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:03:37.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:03:37.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:03:37.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:03:37.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:03:37.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:03:37.281 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:03:37.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:03:37.282 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:03:37.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:03:37.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:03:37.284 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:03:37.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:03:37.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:03:37.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:03:37.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:03:37.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:03:37.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:03:37.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:03:37.310 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:03:37.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:37.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:37.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-17T18:03:37.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:03:37.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:03:37.315 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:03:37.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:03:37.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:03:37.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:03:37.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:03:37.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:03:37.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:03:37.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:03:37.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:03:37.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:03:37.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:03:37.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:03:37.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:03:37.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:03:37.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:37.323 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:37.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:03:37.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:03:37.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:03:37.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-17T18:03:37.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:03:37.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:37.325 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:37.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33174 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-17T18:03:37.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-17T18:03:37.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-17T18:03:37.328 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-17T18:03:37.328 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-17T18:03:37.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-17T18:03:37.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:37.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:37.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:03:37.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:03:37.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:03:37.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:03:37.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:03:37.990 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:03:37.992 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get fsid 2024-09-17T18:03:38.078 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:03:38.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:03:38.080 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get mon_host 2024-09-17T18:03:38.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-17T18:03:38.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-17T18:03:38.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:03:38.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:03:38.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:03:38.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-17T18:03:38.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:03:39.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:39.012 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:03:39.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:03:39.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-17T18:03:39.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-17T18:03:39.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:03:39.051 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:03:39.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5e4d3eb8-851d-426f-a73e-3a10088dbdc4' 2024-09-17T18:03:39.053 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:03:39.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:03:39.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7xOlmbgg/BBAAb86+TM1JzDqAinsbzpBPRw== 2024-09-17T18:03:39.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7xOlmbgg/BBAAb86+TM1JzDqAinsbzpBPRw=="}' 2024-09-17T18:03:39.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 -i td/divergent-priors/0/new.json 2024-09-17T18:03:39.339 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:39.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-17T18:03:39.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7xOlmbgg/BBAAb86+TM1JzDqAinsbzpBPRw== --osd-uuid 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:03:39.386 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:39.384+0000 7f741f818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:39.407 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:39.405+0000 7f741f818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:39.408 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:39.406+0000 7f741f818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:39.409 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:39.406+0000 7f741f818640 -1 bdev(0x55b4853b3c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:03:39.409 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:39.406+0000 7f741f818640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-17T18:03:45.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-17T18:03:45.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:03:45.099 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:03:45.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:03:45.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:03:45.643 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:03:45.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:03:45.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:03:45.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:03:45.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:03:45.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:03:45.679 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:45.676+0000 7f98e47c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:45.681 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:45.679+0000 7f98e47c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:45.682 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:45.680+0000 7f98e47c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:03:46.066 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:03:46.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:46.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:46.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:47.280 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:47.278+0000 7f98e47c1640 -1 Falling back to public interface 2024-09-17T18:03:47.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:47.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:47.493 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:03:47.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:03:47.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:47.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:47.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:48.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:48.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:48.901 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:03:48.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:03:48.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:48.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:49.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:49.938 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:03:49.936+0000 7f98e47c1640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:03:50.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:50.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:50.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:03:50.328 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:03:50.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:50.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:50.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:51.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:51.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:51.878 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:03:51.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:03:51.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:51.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:52.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:53.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:53.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:53.275 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:03:53.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:03:53.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:53.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:53.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:54.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:54.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:54.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:03:54.863 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:03:54.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:54.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:55.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:56.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:56.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:56.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:03:56.266 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:03:56.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:56.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:56.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:57.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:57.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:57.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:03:57.782 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:03:57.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:57.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:58.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:03:59.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:03:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:03:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:03:59.197 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:03:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:03:59.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:03:59.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:00.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:00.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:00.672 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:04:00.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:04:00.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:00.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:01.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:02.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:02.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:02.215 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:04:02.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:04:02.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:02.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:02.723 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:03.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:04:03.725 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:04:03.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:03.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:04.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:05.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:05.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:05.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:04:05.304 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:04:05.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:05.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:06.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:07.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:07.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:07.240 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:04:07.241 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:04:07.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:07.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:08.039 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:09.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:09.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:09.040 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:04:09.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:04:09.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:09.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:09.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:10.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:10.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:10.879 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:04:10.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:04:10.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:10.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:11.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:12.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:12.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:12.413 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:04:12.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:04:12.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:12.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:04:12.833 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2618236074,v1:127.0.0.1:6803/2618236074] [v2:127.0.0.1:6804/2618236074,v1:127.0.0.1:6805/2618236074] exists,up 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:04:12.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:04:12.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:04:12.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:04:12.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-17T18:04:12.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:04:12.843 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:04:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:04:12.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:04:12.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:04:12.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:04:12.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:04:12.848 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:04:12.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:04:12.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:04:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fa7bc93c-a136-4a32-920f-60afc691be80' 2024-09-17T18:04:12.852 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:04:12.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCcxOlm0rA2MxAA9ymoXiU+z4XuUuaaRcoM8A== 2024-09-17T18:04:12.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCcxOlm0rA2MxAA9ymoXiU+z4XuUuaaRcoM8A=="}' 2024-09-17T18:04:12.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fa7bc93c-a136-4a32-920f-60afc691be80 -i td/divergent-priors/1/new.json 2024-09-17T18:04:13.443 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:04:13.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-17T18:04:13.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcxOlm0rA2MxAA9ymoXiU+z4XuUuaaRcoM8A== --osd-uuid fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:04:13.492 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:13.490+0000 7f5dfc0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:13.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:13.492+0000 7f5dfc0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:13.494 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:13.493+0000 7f5dfc0b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:13.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:13.494+0000 7f5dfc0b6640 -1 bdev(0x5568d6a7fc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:04:13.495 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:13.494+0000 7f5dfc0b6640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-17T18:04:18.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-17T18:04:18.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:04:18.888 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:04:18.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:04:18.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:04:19.456 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:04:19.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:04:19.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:04:19.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:04:19.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:04:19.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:04:19.492 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:19.490+0000 7fa5f3c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:19.503 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:19.501+0000 7fa5f3c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:19.504 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:19.503+0000 7fa5f3c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:04:19.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:19.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:04:19.909 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:04:19.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:19.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:20.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:21.099 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:21.097+0000 7fa5f3c20640 -1 Falling back to public interface 2024-09-17T18:04:21.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:21.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:21.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:04:21.352 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:04:21.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:21.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:21.786 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:22.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:22.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:22.787 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:04:22.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:04:22.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:22.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:23.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:24.015 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:24.012+0000 7fa5f3c20640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:04:24.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:24.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:24.191 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:04:24.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:04:24.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:24.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:24.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:25.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:25.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:25.742 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:04:25.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:04:25.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:25.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:26.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:27.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:27.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:27.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:04:27.174 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:04:27.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:27.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:27.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:28.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:28.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:28.713 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:04:28.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:04:28.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:28.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:29.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:30.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:30.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:30.211 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:04:30.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:04:30.211 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:30.212 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:30.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:31.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:31.793 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:04:31.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:31.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:04:31.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:31.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:32.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:33.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:33.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:33.264 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:04:33.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:04:33.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:33.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:33.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:34.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:34.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:34.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:04:34.799 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:04:34.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:34.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:35.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:36.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:36.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:36.221 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:04:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:04:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:36.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:37.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:37.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:37.804 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:04:37.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:04:37.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:37.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:38.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:39.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:39.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:39.510 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:04:39.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:04:39.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:39.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:40.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:41.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:41.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:41.113 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:04:41.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:04:41.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:41.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:41.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:42.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:42.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:42.957 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:04:42.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:04:42.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:42.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:43.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:44.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:44.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:44.411 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:04:44.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:04:44.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:44.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:45.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:46.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:46.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:46.623 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:04:46.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:04:46.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:46.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:04:47.104 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4017056895,v1:127.0.0.1:6811/4017056895] [v2:127.0.0.1:6812/4017056895,v1:127.0.0.1:6813/4017056895] exists,up fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:04:47.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:04:47.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:04:47.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 aea39a78-3527-4ca1-bdc6-c21fd21fb07d' 2024-09-17T18:04:47.111 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:04:47.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:04:47.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/xOlmMFfFBxAA19imioeHuCb31t0SG2tHVg== 2024-09-17T18:04:47.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/xOlmMFfFBxAA19imioeHuCb31t0SG2tHVg=="}' 2024-09-17T18:04:47.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aea39a78-3527-4ca1-bdc6-c21fd21fb07d -i td/divergent-priors/2/new.json 2024-09-17T18:04:47.581 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:04:47.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-17T18:04:47.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/xOlmMFfFBxAA19imioeHuCb31t0SG2tHVg== --osd-uuid aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:04:47.630 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:47.627+0000 7f7354c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:47.650 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:47.648+0000 7f7354c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:47.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:47.649+0000 7f7354c16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:47.652 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:47.649+0000 7f7354c16640 -1 bdev(0x561e37247c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:04:47.652 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:47.650+0000 7f7354c16640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-17T18:04:53.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-17T18:04:53.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:04:53.603 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:04:53.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:04:53.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:04:54.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:04:54.159 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:04:54.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:04:54.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:04:54.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:04:54.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:04:54.194 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:54.191+0000 7fce84411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:54.195 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:54.193+0000 7fce84411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:54.196 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:54.194+0000 7fce84411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:54.634 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:04:54.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:04:54.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:54.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:04:55.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:55.543 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:55.542+0000 7fce84411640 -1 Falling back to public interface 2024-09-17T18:04:56.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:56.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:56.094 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:04:56.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:04:56.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:56.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:04:56.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:57.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:57.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:57.529 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:04:57.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:04:57.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:57.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:04:57.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:04:58.357 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:04:58.356+0000 7fce84411640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:04:58.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:04:58.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:04:58.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:04:58.927 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:04:58.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:04:58.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:04:59.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:00.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:00.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:00.454 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:05:00.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:05:00.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:00.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:00.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:01.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:01.869 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:05:01.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:01.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:05:01.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:01.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:02.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:03.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:03.340 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:05:03.341 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:05:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:03.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:03.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:04.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:04.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:04.790 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:05:04.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:05:04.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:04.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:05.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:06.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:06.370 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:05:06.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:05:06.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:06.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:06.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:07.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:07.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:07.796 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:05:07.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:05:07.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:07.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:08.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:09.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:09.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:09.343 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:05:09.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:05:09.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:09.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:09.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:10.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:10.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:10.814 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:05:10.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:05:10.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:10.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:11.341 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:12.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:12.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:12.342 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:05:12.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:05:12.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:12.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:13.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:14.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:14.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:14.246 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:05:14.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:05:14.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:14.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:15.026 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:16.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:16.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:16.028 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:05:16.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:05:16.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:16.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:16.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:17.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:17.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:17.481 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:05:17.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:05:17.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:17.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:18.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:19.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:19.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:19.587 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:05:19.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:05:19.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:19.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:20.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:05:21.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:05:21.391 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:05:21.391 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:05:21.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:05:21.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:05:21.392 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:05:21.894 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2980874150,v1:127.0.0.1:6819/2980874150] [v2:127.0.0.1:6820/2980874150,v1:127.0.0.1:6821/2980874150] exists,up aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:05:21.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:05:21.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:05:21.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:05:21.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-17T18:05:22.491 INFO:tasks.workunit.client.0.smithi157.stderr:noout is set 2024-09-17T18:05:22.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-17T18:05:23.204 INFO:tasks.workunit.client.0.smithi157.stderr:noin is set 2024-09-17T18:05:23.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-17T18:05:23.958 INFO:tasks.workunit.client.0.smithi157.stderr:nodown is set 2024-09-17T18:05:23.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-17T18:05:23.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-17T18:05:24.948 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-17T18:05:24.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:05:25.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-17T18:05:26.637 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 size to 3 2024-09-17T18:05:26.649 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-17T18:05:27.375 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 2 2024-09-17T18:05:27.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-17T18:05:27.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:05:27.388 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:05:27.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:05:27.777 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:05:27.777 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:05:27.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:05:27.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:27.777 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:05:27.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-17T18:05:27.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-17T18:05:27.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-17T18:05:27.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:27.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:05:28.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-17T18:05:28.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-17T18:05:28.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672972' 2024-09-17T18:05:28.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:28.022 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:05:28.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-17T18:05:28.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-17T18:05:28.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672972 2-64424509449' 2024-09-17T18:05:28.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:28.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-17T18:05:28.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:28.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:05:28.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-17T18:05:28.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:28.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-17T18:05:28.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-17T18:05:28.146 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836502 2024-09-17T18:05:28.146 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:28.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836502 2024-09-17T18:05:28.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:05:29.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:05:29.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:29.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-17T18:05:29.979 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:29.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-17T18:05:29.980 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:29.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:05:29.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-17T18:05:29.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:29.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-17T18:05:29.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-17T18:05:29.984 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672972 2024-09-17T18:05:29.984 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:05:30.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-17T18:05:30.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:30.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-17T18:05:30.431 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:30.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:05:30.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-17T18:05:30.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:30.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-17T18:05:30.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-17T18:05:30.433 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509449 2024-09-17T18:05:30.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:05:30.837 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:05:30.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:05:30.838 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:05:30.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:05:30.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:05:30.838 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:05:30.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:05:30.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:05:30.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:05:30.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:05:30.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:05:30.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:05:31.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:05:31.354 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:05:31.354 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:05:31.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:05:31.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:31.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:05:31.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-17T18:05:31.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-17T18:05:31.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-17T18:05:31.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:31.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:05:31.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-17T18:05:31.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-17T18:05:31.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-09-17T18:05:31.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:31.595 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509451' 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-17T18:05:31.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:31.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:05:31.718 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-17T18:05:31.718 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:31.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-17T18:05:31.719 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836504 2024-09-17T18:05:31.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-17T18:05:31.719 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:32.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-17T18:05:32.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:05:33.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:05:33.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:33.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-17T18:05:33.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:33.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-17T18:05:33.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:33.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:05:33.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-17T18:05:33.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:33.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-17T18:05:33.590 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672974 2024-09-17T18:05:33.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-17T18:05:33.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:05:34.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-17T18:05:34.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:34.013 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T18:05:34.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:34.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:05:34.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T18:05:34.015 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:34.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T18:05:34.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T18:05:34.016 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509451 2024-09-17T18:05:34.016 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:05:34.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-17T18:05:34.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:05:34.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:05:34.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:05:34.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:05:34.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:05:34.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:05:34.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:05:34.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:05:34.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:05:34.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:05:34.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:05:35.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:05:35.353 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:05:35.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:05:35.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:05:35.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:05:35.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:05:35.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:05:35.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-17T18:05:35.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-17T18:05:36.276 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:05:36.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-17T18:05:36.287 INFO:tasks.workunit.client.0.smithi157.stdout:primary and soon to be divergent is 1 2024-09-17T18:05:36.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-17T18:05:36.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:05:36.661 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:05:36.661 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:05:25.725181+0000 0'0 29:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:43:16.938812+0000 0 0 2024-09-17T18:05:36.661 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:05:36.661 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:05:36.661 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-17T18:05:36.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-17T18:05:36.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-17T18:05:36.674 INFO:tasks.workunit.client.0.smithi157.stdout:writing initial objects 2024-09-17T18:05:36.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-17T18:05:36.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:36.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-17T18:05:36.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:36.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-17T18:05:36.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:36.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-17T18:05:37.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-17T18:05:37.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-17T18:05:37.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-17T18:05:37.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-17T18:05:37.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-17T18:05:37.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-17T18:05:37.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:37.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-17T18:05:38.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-17T18:05:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-17T18:05:38.388 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.389 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-17T18:05:38.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-17T18:05:38.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-17T18:05:38.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-17T18:05:38.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:38.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-17T18:05:39.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-17T18:05:39.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-17T18:05:39.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-17T18:05:39.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-17T18:05:39.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-17T18:05:39.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-17T18:05:39.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:39.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-17T18:05:40.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-17T18:05:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-17T18:05:40.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-17T18:05:40.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-17T18:05:40.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-17T18:05:40.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-17T18:05:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:40.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-17T18:05:41.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-17T18:05:41.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-17T18:05:41.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-17T18:05:41.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-17T18:05:41.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-17T18:05:41.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:41.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-17T18:05:42.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-17T18:05:42.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-17T18:05:42.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-17T18:05:42.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-17T18:05:42.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-17T18:05:42.706 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-17T18:05:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-17T18:05:42.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:42.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-17T18:05:43.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-17T18:05:43.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-17T18:05:43.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-17T18:05:43.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-17T18:05:43.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-17T18:05:43.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-17T18:05:43.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:43.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-17T18:05:44.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-17T18:05:44.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.266 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-17T18:05:44.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-17T18:05:44.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-17T18:05:44.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-17T18:05:44.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:44.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-17T18:05:45.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:45.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-17T18:05:45.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:45.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-17T18:05:45.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:45.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-17T18:05:45.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:45.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-17T18:05:45.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:45.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-17T18:05:46.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-17T18:05:46.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-17T18:05:46.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-17T18:05:46.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-17T18:05:46.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-17T18:05:46.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-17T18:05:46.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:46.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-17T18:05:47.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-17T18:05:47.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-17T18:05:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-17T18:05:47.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-17T18:05:47.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-17T18:05:47.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-17T18:05:47.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:47.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-17T18:05:48.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.071 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-17T18:05:48.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-17T18:05:48.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-17T18:05:48.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-17T18:05:48.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-17T18:05:48.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-17T18:05:48.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:48.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-17T18:05:49.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-17T18:05:49.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-17T18:05:49.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-17T18:05:49.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-17T18:05:49.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-17T18:05:49.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-17T18:05:49.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:49.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-17T18:05:50.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-17T18:05:50.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-17T18:05:50.375 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-17T18:05:50.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-17T18:05:50.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-17T18:05:50.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-17T18:05:50.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:50.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-17T18:05:51.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:51.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-17T18:05:51.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-17T18:05:51.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:05:51.416 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:05:51.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:05:51.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:05:51.417 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:05:51.470 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:05:51.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:05:51.902 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:05:51.902 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:05:51.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:05:51.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:51.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:05:52.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-17T18:05:52.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-17T18:05:52.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-17T18:05:52.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:52.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:05:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-17T18:05:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-17T18:05:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672980' 2024-09-17T18:05:52.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:05:52.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:05:52.264 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-42949672980 2-64424509458' 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:52.265 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:05:52.266 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-17T18:05:52.266 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:52.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-17T18:05:52.267 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836510 2024-09-17T18:05:52.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-17T18:05:52.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:52.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836510 2024-09-17T18:05:52.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:05:53.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:05:53.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:05:54.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-17T18:05:54.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:54.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-17T18:05:54.130 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:54.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:05:54.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-17T18:05:54.131 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:54.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-17T18:05:54.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-17T18:05:54.133 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672980 2024-09-17T18:05:54.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:05:54.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-17T18:05:54.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:05:54.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-17T18:05:54.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:05:54.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:05:54.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-17T18:05:54.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:05:54.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-17T18:05:54.567 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-17T18:05:54.568 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509458 2024-09-17T18:05:54.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:05:54.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-17T18:05:54.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:05:54.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:05:54.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:05:55.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:05:55.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:05:55.920 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:05:55.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:05:55.921 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-17T18:05:56.452 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-17T18:05:56.453 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-17T18:05:56.453 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:05:56.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-17T18:05:56.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-17T18:05:56.913 INFO:tasks.workunit.client.0.smithi157.stdout:blackholing osds 0 2 2024-09-17T18:05:56.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:05:57.288 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:05:57.288 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-17T18:05:25.725181+0000 29'100 29:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:43:16.938812+0000 0 0 2024-09-17T18:05:57.288 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:05:57.288 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:05:57.288 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:05:57.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:05:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-17T18:05:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:05:57.300 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:05:57.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-17T18:05:57.378 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:05:57.378 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:05:57.378 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:05:57.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:05:57.386 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:05:57.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:05:57.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-17T18:05:57.463 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:05:57.463 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:05:57.463 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:05:57.470 INFO:tasks.workunit.client.0.smithi157.stdout:create new divergent objects 2024-09-17T18:05:57.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-17T18:05:57.471 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-17T18:05:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:05:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:05:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-17T18:05:57.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-17T18:05:57.472 INFO:tasks.workunit.client.0.smithi157.stdout:writing divergent objects 2024-09-17T18:05:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-17T18:05:57.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-17T18:05:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-17T18:05:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-17T18:05:57.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-17T18:05:57.474 INFO:tasks.workunit.client.0.smithi157.stdout:remove divergent objects 2024-09-17T18:05:57.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-17T18:05:57.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-17T18:05:57.474 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-17T18:05:57.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-17T18:05:57.475 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-17T18:05:57.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-17T18:05:57.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-17T18:05:57.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-17T18:05:57.477 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-17T18:05:57.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-17T18:05:57.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-17T18:05:57.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-17T18:06:07.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-17T18:06:07.486 INFO:tasks.workunit.client.0.smithi157.stderr:rados: no process found 2024-09-17T18:06:07.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-17T18:06:07.486 INFO:tasks.workunit.client.0.smithi157.stdout:killing all the osds 2024-09-17T18:06:07.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:06:07.866 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:06:07.867 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 active+clean 2024-09-17T18:05:25.725181+0000 29'106 29:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:43:16.938812+0000 0 0 2024-09-17T18:06:07.868 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:06:07.868 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:06:07.868 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:06:07.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-17T18:06:07.878 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:06:07.878 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:06:07.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:06:07.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:06:07.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:06:07.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:06:07.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-17T18:06:07.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-17T18:06:08.471 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:06:08.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-17T18:06:08.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-17T18:06:09.024 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:06:09.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-17T18:06:09.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-17T18:06:09.620 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:06:09.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:09.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-17T18:06:10.146 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already out. 2024-09-17T18:06:10.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:10.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-17T18:06:10.664 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already out. 2024-09-17T18:06:10.676 INFO:tasks.workunit.client.0.smithi157.stdout:bringing up non_divergent 0 2 2024-09-17T18:06:10.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-17T18:06:10.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:06:11.086 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:06:11.086 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 stale+active+clean 2024-09-17T18:05:25.725181+0000 29'106 29:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:43:16.938812+0000 0 0 2024-09-17T18:06:11.086 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:06:11.086 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:06:11.086 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:06:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:06:11.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:06:11.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:06:11.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:06:11.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:06:11.103 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:06:11.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:06:11.134 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:11.133+0000 7f3d7461d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:11.136 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:11.135+0000 7f3d7461d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:11.137 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:11.136+0000 7f3d7461d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:11.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:06:11.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:06:11.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:06:11.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:06:11.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:06:11.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:11.538 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:06:11.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:06:11.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:11.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:06:11.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:11.999 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:11.998+0000 7f3d7461d640 -1 Falling back to public interface 2024-09-17T18:06:12.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:12.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:12.964 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:06:12.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:06:12.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:12.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:06:13.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:14.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:14.408 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:14.408 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:06:14.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:06:14.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:14.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:06:14.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:15.730 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:15.728+0000 7f3d7461d640 -1 osd.0 29 log_to_monitors true 2024-09-17T18:06:15.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:15.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:15.813 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:06:15.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:06:15.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:15.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:06:16.290 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3184318350,v1:127.0.0.1:6803/3184318350] [v2:127.0.0.1:6804/3184318350,v1:127.0.0.1:6805/3184318350] exists,up 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-17T18:06:16.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:06:16.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:06:16.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:06:16.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:06:16.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:06:16.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:06:16.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:06:16.297 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:06:16.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:06:16.328 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:16.326+0000 7f9ea514d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:16.330 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:16.328+0000 7f9ea514d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:16.331 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:16.329+0000 7f9ea514d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:06:16.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:06:16.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:06:16.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:06:16.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:06:16.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:06:16.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:16.716 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:06:16.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:06:16.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:16.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:06:17.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:17.689 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:17.684+0000 7f9ea514d640 -1 Falling back to public interface 2024-09-17T18:06:18.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:18.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:18.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:06:18.137 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:06:18.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:18.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:06:18.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:19.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:19.563 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:06:19.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:19.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:06:19.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:19.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:06:19.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:06:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:06:20.906+0000 7f9ea514d640 -1 osd.2 29 log_to_monitors true 2024-09-17T18:06:20.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:06:20.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:06:20.998 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:06:20.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:06:20.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:06:20.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:06:21.405 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/62858284,v1:127.0.0.1:6811/62858284] [v2:127.0.0.1:6812/62858284,v1:127.0.0.1:6813/62858284] exists,up aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:06:21.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:06:21.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:06:21.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:06:21.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:21.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-17T18:06:21.985 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already in. 2024-09-17T18:06:21.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:06:21.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-17T18:06:22.569 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already in. 2024-09-17T18:06:22.581 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:06:22.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-17T18:06:22.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:06:22.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:06:22.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:06:22.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:06:22.583 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:06:22.583 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:06:22.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:06:22.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:06:22.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:06:22.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:06:22.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:06:22.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:06:22.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:06:22.637 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:06:22.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:06:23.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:06:23.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-17T18:06:23.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-17T18:06:23.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-17T18:06:23.310 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:06:23.310 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:06:23.397 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:06:23.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:06:23.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:06:23.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:06:23.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:06:23.401 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:06:23.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-17T18:06:23.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-17T18:06:23.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-17T18:06:23.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:06:23.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-17T18:06:23.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:06:23.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:06:23.522 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-17T18:06:23.522 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:06:23.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-17T18:06:23.523 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 154618822660 2024-09-17T18:06:23.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-17T18:06:23.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:06:23.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-17T18:06:23.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:06:24.918 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:06:24.919 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:06:25.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-17T18:06:25.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:06:25.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-17T18:06:25.346 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:06:25.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:06:25.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-17T18:06:25.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:06:25.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-17T18:06:25.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-17T18:06:25.348 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 163208757251 2024-09-17T18:06:25.349 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:06:25.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-17T18:06:25.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:06:25.806 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:25.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:06:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:26.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:26.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:26.745 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:26.745 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:26.745 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:27.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:27.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:06:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:06:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:06:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:27.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:27.376 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:27.377 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:27.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:27.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:27.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:27.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:28.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:28.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:28.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:28.283 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:28.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:28.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:28.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:06:28.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:28.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:06:29.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:29.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:29.019 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:29.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:29.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:29.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:29.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:29.020 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:29.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:29.407 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:29.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:29.408 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:29.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:06:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:30.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:06:30.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:30.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:30.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:30.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:30.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:30.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:30.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:30.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:31.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:31.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:31.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:31.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:31.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:31.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:32.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:32.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:32.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:06:32.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:32.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:06:33.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:33.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:33.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:33.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:33.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:33.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:33.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:33.181 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:33.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:33.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:33.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:33.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:34.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:34.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:34.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:34.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:34.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:06:34.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:34.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:06:36.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:36.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:36.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:36.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:36.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:36.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:36.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:36.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:36.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:36.661 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:36.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:36.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:37.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:37.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:37.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:37.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:37.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:37.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:06:37.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:37.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:06:40.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:40.922 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:40.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:40.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:40.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:40.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:40.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:40.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:41.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:41.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:41.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:41.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:41.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:41.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:41.865 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:42.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:42.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:42.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:06:42.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:42.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:48.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:48.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:49.224 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:49.224 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:49.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:49.225 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:49.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:06:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:06:50.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:06:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:06:57.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:06:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:06:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:06:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:06:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:06:57.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:06:57.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:06:57.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:06:57.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:06:57.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:06:57.991 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:06:58.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:06:58.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:06:58.521 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:06:59.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:06:59.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:06:59.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:06:59.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:06:59.605 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "7c0dd7cd-4043-46f2-a272-b1cfba543978", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:06:59.603235+0000", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:06:59.606 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.607 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:06:59.608 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.609 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.610 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.611 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:06:59.612 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:03:37.357537Z", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:03:37.357537Z", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:06:59.613 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:06:59.614 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.615 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:06:59.616 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 41, 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:03:38.243071+0000", 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:06:22.776011+0000", 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:06:21.147389+0000", 2024-09-17T18:06:59.617 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:06:22.361525+0000", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 9, 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:06:59.618 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:06:59.619 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:06:59.620 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:06:59.620 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:06:59.620 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.620 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:05:24.271662+0000", 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 1, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool", 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:06:59.621 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 1, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 1, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 1, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 1, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 1, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:06:59.622 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "29", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 0, 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 0, 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-17T18:06:59.623 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:06:59.624 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:06:59.625 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {}, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "read_balance": { 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "score_type": "Fair distribution", 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "score_acting": 2.9900000095367432, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "score_stable": 2.9900000095367432, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_score": 0.67000001668930054, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_acting": 2, 2024-09-17T18:06:59.626 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_stable": 2, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity_weighted": 1, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity": 1, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity_weighted": 1 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "5e4d3eb8-851d-426f-a73e-3a10088dbdc4", 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:06:59.627 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 36, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 40, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 30, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:06:59.629 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.630 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.631 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/3184318350", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/3184318350", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3184318350", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3184318350", 2024-09-17T18:06:59.632 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "fa7bc93c-a136-4a32-920f-60afc691be80", 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:06:59.633 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 23, 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 31, 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.634 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.635 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.636 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.637 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.638 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/4017056895", 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/4017056895", 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4017056895", 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4017056895", 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:06:59.639 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "aea39a78-3527-4ca1-bdc6-c21fd21fb07d", 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 38, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 0, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 32, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:06:59.640 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.641 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.642 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.643 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:06:59.644 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/62858284", 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/62858284", 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62858284", 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62858284", 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:06:59.645 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:08.313824+0000", 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:04:11.041467+0000", 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:06:59.646 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:08.829427+0000", 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:04:46.569875+0000", 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.647 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:09.382789+0000", 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:05:21.135184+0000", 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:06:59.648 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:06:59.649 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.650 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/3184318350,v1:127.0.0.1:6805/3184318350]", 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:06:59.651 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:06:59.652 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:06:59.653 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:03:44.448295Z", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:06:59.654 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/3184318350,v1:127.0.0.1:6803/3184318350]", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3184318350,v1:127.0.0.1:6809/3184318350]", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3184318350,v1:127.0.0.1:6807/3184318350]", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:06:59.655 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:06:59.656 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/4017056895,v1:127.0.0.1:6813/4017056895]", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:06:59.657 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:06:59.658 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:06:59.659 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:06:59.659 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.660 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:04:18.248921Z", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/4017056895,v1:127.0.0.1:6811/4017056895]", 2024-09-17T18:06:59.661 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4017056895,v1:127.0.0.1:6817/4017056895]", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4017056895,v1:127.0.0.1:6815/4017056895]", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:06:59.662 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/62858284,v1:127.0.0.1:6813/62858284]", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:06:59.663 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.664 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:04:52.924630Z", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/62858284,v1:127.0.0.1:6811/62858284]", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62858284,v1:127.0.0.1:6817/62858284]", 2024-09-17T18:06:59.665 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62858284,v1:127.0.0.1:6815/62858284]", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:06:59.666 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 36 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:06:59.667 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 39 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 41, 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:06:59.668 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.669 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.670 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.671 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:06:59.672 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.673 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:06:59.674 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.675 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:06:59.676 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.677 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:06:59.678 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:06:59.679 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.680 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:06:59.681 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:06:59.682 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:06:59.683 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.684 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:03:38:218545+0000", 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:06:59.685 INFO:tasks.workunit.client.0.smithi157.stderr:report 3774533780 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:06:59.686 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:06:59.687 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 1, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48300, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:06:59.688 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 100, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 100, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:06:59.689 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:06:59.690 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:06:59.691 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 16, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 16, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:06:59.692 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83656, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2104, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:06:59.693 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489144, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036883456, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2154496, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 418637, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10397, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416931 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.694 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:06:59.695 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83656, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2104, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:06:59.696 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489144, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036883456, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2154496, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 418637, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10397, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416931 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.697 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:06:59.698 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48300, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:06:59.699 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 100, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 100, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:06:59.700 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:06:59.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1220608, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 143276, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:06:59.702 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 16, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 16, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 154618822667 2024-09-17T18:06:59.703 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672984 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 163208757258 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 1 2024-09-17T18:06:59.704 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 1, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:06:59.705 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:06:59.706 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 4, 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:05:48.947083+0000", 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "services": { 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": { 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "daemons": { 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": "", 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "2": { 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "start_epoch": 0, 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "start_stamp": "0.000000", 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "gid": 0, 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "metadata": {}, 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: "task_status": {} 2024-09-17T18:06:59.707 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 90, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 379, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 516, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:06:59.708 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:06:59.709 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:06:59.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:06:59.709 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-17T18:06:59.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-17T18:06:59.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-17T18:06:59.710 INFO:tasks.workunit.client.0.smithi157.stdout:writing non-divergent object existing_4 2024-09-17T18:06:59.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:00.002 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:00.002 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 29'100 41:132 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:00.002 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:00.002 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:00.003 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:00.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:07:00.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:07:00.107 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:07:00.107 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:07:00.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:07:00.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:07:00.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:07:00.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:07:00.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:07:00.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:07:00.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:07:00.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:07:00.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:07:00.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:07:00.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822669 2024-09-17T18:07:00.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822669 2024-09-17T18:07:00.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822669' 2024-09-17T18:07:00.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:07:00.714 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:07:00.803 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:07:00.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:07:00.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:07:00.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:07:00.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:07:00.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757260 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757260 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822669 2-163208757260' 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822669 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:07:00.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:07:00.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822669 2024-09-17T18:07:00.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:07:00.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822669 2024-09-17T18:07:00.929 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 154618822669 2024-09-17T18:07:00.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2024-09-17T18:07:00.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:07:01.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822670 -lt 154618822669 2024-09-17T18:07:01.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:07:01.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757260 2024-09-17T18:07:01.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:07:01.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:07:01.422 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757260 2024-09-17T18:07:01.423 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:07:01.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757260 2024-09-17T18:07:01.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2024-09-17T18:07:01.424 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 163208757260 2024-09-17T18:07:01.424 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:07:01.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757260 -lt 163208757260 2024-09-17T18:07:01.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:07:01.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:01.849 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:02.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:02.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:02.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:02.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:02.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:03.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:03.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:07:03.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:07:03.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:07:03.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:03.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:07:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:03.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:03.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:03.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:03.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:03.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:03.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:03.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:03.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:03.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:03.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:03.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:04.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:04.383 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:04.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:04.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:04.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:07:04.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:04.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:07:05.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:05.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:05.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:05.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:05.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:05.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:05.525 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:06.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:06.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:06.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:06.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:06.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:06.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:06.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:07:06.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:06.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:07:06.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:06.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:06.986 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:06.986 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:06.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:06.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:06.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:06.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:07.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:07.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:07.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:07.375 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:07.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:07.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:08.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:08.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:08.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:07:08.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:08.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:07:09.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:09.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:09.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:09.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:09.651 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:09.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:09.652 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:10.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:10.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:10.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:10.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:10.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:10.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:10.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:07:10.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:10.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:07:12.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:12.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:12.340 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:12.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:12.732 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:12.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:12.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:13.267 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:13.268 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:13.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:13.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:13.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:07:13.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:13.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:07:17.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:17.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:17.009 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:17.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:17.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:17.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:17.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:17.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:17.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:17.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:17.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:17.411 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:17.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:17.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:17.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:17.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:17.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:17.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:17.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:17.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:17.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:18.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:18.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:18.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:07:18.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:18.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:24.885 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:24.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:25.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:25.278 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:25.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:25.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:25.807 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:07:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:07:26.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:07:33.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:07:33.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:07:33.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:07:33.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:07:33.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:07:33.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:07:33.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:07:33.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:07:34.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:07:34.028 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:07:34.029 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:07:34.029 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:07:34.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:07:34.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:07:34.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:07:34.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:07:34.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:07:34.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:07:34.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:07:34.561 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:07:34.561 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:07:35.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:07:35.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:07:35.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:07:35.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "7c0dd7cd-4043-46f2-a272-b1cfba543978", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:07:35.620604+0000", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:07:35.623 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:07:35.624 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.625 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 2 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:07:35.626 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:07:35.627 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.628 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:07:35.629 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:03:37.357537Z", 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:03:37.357537Z", 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:07:35.630 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:07:35.631 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:07:35.632 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:07:35.633 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:07:35.634 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 41, 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:03:38.243071+0000", 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:06:22.776011+0000", 2024-09-17T18:07:35.635 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:06:21.147389+0000", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:06:22.361525+0000", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.636 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 9, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.637 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:05:24.271662+0000", 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 1, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool", 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:07:35.638 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 1, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 1, 2024-09-17T18:07:35.639 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 1, 2024-09-17T18:07:35.640 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 1, 2024-09-17T18:07:35.640 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 1, 2024-09-17T18:07:35.640 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "29", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:07:35.641 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:07:35.642 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:07:35.643 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {}, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "read_balance": { 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "score_type": "Fair distribution", 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "score_acting": 2.9900000095367432, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "score_stable": 2.9900000095367432, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_score": 0.67000001668930054, 2024-09-17T18:07:35.644 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_acting": 2, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_stable": 2, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity_weighted": 1, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity": 1, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity_weighted": 1 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "5e4d3eb8-851d-426f-a73e-3a10088dbdc4", 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:07:35.645 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 36, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 40, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 30, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:07:35.646 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.648 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.649 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 3184318350 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/3184318350", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/3184318350", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3184318350", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3184318350", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:07:35.650 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "fa7bc93c-a136-4a32-920f-60afc691be80", 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:07:35.651 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 23, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 31, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.652 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.653 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.654 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.655 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4017056895 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/4017056895", 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/4017056895", 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4017056895", 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4017056895", 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.656 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "aea39a78-3527-4ca1-bdc6-c21fd21fb07d", 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 38, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 0, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 32, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:07:35.657 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:07:35.658 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:07:35.659 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:07:35.660 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 62858284 2024-09-17T18:07:35.661 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/62858284", 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/62858284", 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62858284", 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62858284", 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:08.313824+0000", 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:04:11.041467+0000", 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:07:35.663 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:08.829427+0000", 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:07:35.664 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:04:46.569875+0000", 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:06:09.382789+0000", 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:07:35.665 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:05:21.135184+0000", 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:07:35.666 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:07:35.667 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:07:35.668 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:07:35.669 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/3184318350,v1:127.0.0.1:6805/3184318350]", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:07:35.670 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:03:44.448295Z", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:07:35.671 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/3184318350,v1:127.0.0.1:6803/3184318350]", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3184318350,v1:127.0.0.1:6809/3184318350]", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3184318350,v1:127.0.0.1:6807/3184318350]", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:07:35.672 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:07:35.673 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/4017056895,v1:127.0.0.1:6813/4017056895]", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:07:35.674 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:04:18.248921Z", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:07:35.675 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/4017056895,v1:127.0.0.1:6811/4017056895]", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4017056895,v1:127.0.0.1:6817/4017056895]", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4017056895,v1:127.0.0.1:6815/4017056895]", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:07:35.676 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/62858284,v1:127.0.0.1:6813/62858284]", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:07:35.677 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:07:35.678 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:04:52.924630Z", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/62858284,v1:127.0.0.1:6811/62858284]", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62858284,v1:127.0.0.1:6817/62858284]", 2024-09-17T18:07:35.679 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62858284,v1:127.0.0.1:6815/62858284]", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.680 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 36 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:07:35.681 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 39 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 41, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:07:35.682 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.683 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:07:35.684 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.685 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:07:35.686 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.687 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:07:35.688 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.689 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:07:35.690 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.691 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.692 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:07:35.693 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.694 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:07:35.695 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:07:35.696 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:07:35.697 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:03:38:218545+0000", 2024-09-17T18:07:35.698 INFO:tasks.workunit.client.0.smithi157.stderr:report 514650404 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:07:35.699 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:07:35.700 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 1, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:07:35.701 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 47971, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:07:35.702 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:07:35.703 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:07:35.704 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 17, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 17, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:07:35.705 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83656, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2104, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489144, 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:07:35.706 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036883456, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2154496, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 417979, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10397, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416931 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:07:35.707 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:07:35.708 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83656, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2104, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489144, 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:07:35.709 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036883456, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2154496, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 417979, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10397, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416931 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:07:35.710 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.711 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 47971, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 100, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 300, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 100, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:07:35.712 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 100, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:07:35.713 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:07:35.714 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1220608, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 142618, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:07:35.715 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 17, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 17, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 2, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 2, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 154618822676 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.716 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672984 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 163208757267 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 1 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.717 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 1, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:07:35.718 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 6, 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:07:22.963730+0000", 2024-09-17T18:07:35.719 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 108, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 445, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 600, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:07:35.720 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:07:35.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:07:35.721 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery 2024-09-17T18:07:35.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-17T18:07:35.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:36.030 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:36.030 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 41:133 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:36.031 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:36.031 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:36.031 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:36.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:07:36.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:07:36.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok set_recovery_delay 100000 2024-09-17T18:07:36.124 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:07:36.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:07:36.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok set_recovery_delay 100000 2024-09-17T18:07:36.214 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:07:36.222 INFO:tasks.workunit.client.0.smithi157.stdout:revive divergent 1 2024-09-17T18:07:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-17T18:07:36.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:36.591 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:36.591 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 41:133 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:36.591 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:36.591 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:36.591 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:36.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-17T18:07:37.228 INFO:tasks.workunit.client.0.smithi157.stderr:noup is set 2024-09-17T18:07:37.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-17T18:07:37.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:07:37.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:07:37.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:07:37.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:07:37.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:07:37.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:07:37.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:07:37.245 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:07:37.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:07:37.278 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:37.275+0000 7fec15214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:37.304 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:37.302+0000 7fec15214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:37.305 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:37.303+0000 7fec15214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:37.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-17T18:07:37.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-17T18:07:38.395 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:38.393+0000 7fec15214640 -1 Falling back to public interface 2024-09-17T18:07:41.765 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:41.762+0000 7fec15214640 -1 osd.1 29 log_to_monitors true 2024-09-17T18:07:42.659 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery divergent 2024-09-17T18:07:42.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-17T18:07:42.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:43.048 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:43.048 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 43:137 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:43.048 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:43.048 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:43.048 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:07:43.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok set_recovery_delay 100000 2024-09-17T18:07:43.143 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:07:43.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-17T18:07:43.741 INFO:tasks.workunit.client.0.smithi157.stderr:noup is unset 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:07:43.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:07:43.754 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:07:43.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:07:43.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:07:44.205 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3184318350,v1:127.0.0.1:6803/3184318350] [v2:127.0.0.1:6804/3184318350,v1:127.0.0.1:6805/3184318350] exists,up 5e4d3eb8-851d-426f-a73e-3a10088dbdc4 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:07:44.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:07:44.207 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:07:44.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:07:44.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:07:44.641 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/235845353,v1:127.0.0.1:6819/235845353] [v2:127.0.0.1:6820/235845353,v1:127.0.0.1:6821/235845353] exists,up fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:07:44.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:07:44.643 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:07:44.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:07:44.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:07:44.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:07:45.221 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/62858284,v1:127.0.0.1:6811/62858284] [v2:127.0.0.1:6812/62858284,v1:127.0.0.1:6813/62858284] exists,up aea39a78-3527-4ca1-bdc6-c21fd21fb07d 2024-09-17T18:07:45.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:07:45.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:07:45.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:07:45.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:45.595 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:45.595 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 43:137 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:45.595 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:45.595 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:45.595 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:45.606 INFO:tasks.workunit.client.0.smithi157.stdout:wait for peering 2024-09-17T18:07:45.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-17T18:07:45.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:45.978 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:45.978 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 43:137 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:45.978 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:45.978 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:45.978 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:45.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-17T18:07:46.135 INFO:tasks.workunit.client.0.smithi157.stdout:killing divergent 1 2024-09-17T18:07:46.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-17T18:07:46.135 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:46.512 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:46.512 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 43:137 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:46.512 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:46.512 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:46.513 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:46.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-17T18:07:46.523 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:07:46.524 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:07:46.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:07:46.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:07:46.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.33174.out 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33174.out 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-17T18:07:46.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33174.out 2024-09-17T18:07:47.982 INFO:tasks.workunit.client.0.smithi157.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-17T18:07:47.983 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-17T18:07:47.983 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-17T18:07:47.991 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-17T18:07:47.991 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-17T18:07:47.991 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-17T18:07:47.998 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-17T18:07:48.000 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-17T18:07:48.007 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:169157fa:::existing_81:head# 2024-09-17T18:07:48.009 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:17731382:::existing_58:head# 2024-09-17T18:07:48.010 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-17T18:07:48.010 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-17T18:07:48.011 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-17T18:07:48.011 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-17T18:07:48.012 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-17T18:07:48.012 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:28566edb:::existing_78:head# 2024-09-17T18:07:48.013 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:2a101182:::existing_20:head# 2024-09-17T18:07:48.013 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-17T18:07:48.014 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-17T18:07:48.016 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-17T18:07:48.016 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-17T18:07:48.016 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-17T18:07:48.016 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-17T18:07:48.017 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-17T18:07:48.017 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-17T18:07:48.017 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-17T18:07:48.018 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-17T18:07:48.018 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:53588912:::existing_9:head# 2024-09-17T18:07:48.019 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-17T18:07:48.020 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-17T18:07:48.021 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-17T18:07:48.021 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-17T18:07:48.021 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:602f83fe:::foo:head# 2024-09-17T18:07:48.022 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-17T18:07:48.022 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:628918ca:::existing_79:head# 2024-09-17T18:07:48.023 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-17T18:07:48.023 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-17T18:07:48.024 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-17T18:07:48.024 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-17T18:07:48.025 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-17T18:07:48.025 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-17T18:07:48.026 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-17T18:07:48.026 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-17T18:07:48.026 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:750ace21:::existing_19:head# 2024-09-17T18:07:48.027 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-17T18:07:48.027 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:79881152:::existing_63:head# 2024-09-17T18:07:48.028 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-17T18:07:48.028 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:81490677:::existing_76:head# 2024-09-17T18:07:48.029 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-17T18:07:48.029 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-17T18:07:48.030 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-17T18:07:48.030 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-17T18:07:48.031 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:92731e71:::existing_89:head# 2024-09-17T18:07:48.031 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:928205cf:::existing_82:head# 2024-09-17T18:07:48.032 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:93be2267:::existing_24:head# 2024-09-17T18:07:48.033 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-17T18:07:48.033 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-17T18:07:48.034 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-17T18:07:48.034 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-17T18:07:48.035 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-17T18:07:48.035 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a504a533:::existing_90:head# 2024-09-17T18:07:48.035 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a691337e:::existing_61:head# 2024-09-17T18:07:48.036 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-17T18:07:48.037 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-17T18:07:48.038 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-17T18:07:48.038 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-17T18:07:48.039 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-17T18:07:48.039 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:b343e277:::existing_88:head# 2024-09-17T18:07:48.040 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-17T18:07:48.040 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-17T18:07:48.041 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-17T18:07:48.042 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-17T18:07:48.042 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:bebec877:::existing_74:head# 2024-09-17T18:07:48.043 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-17T18:07:48.043 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:c156033e:::existing_100:head# 2024-09-17T18:07:48.043 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-17T18:07:48.044 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-17T18:07:48.045 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-17T18:07:48.045 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-17T18:07:48.045 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-17T18:07:48.046 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-17T18:07:48.047 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-17T18:07:48.047 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-17T18:07:48.048 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-17T18:07:48.048 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-17T18:07:48.049 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-17T18:07:48.049 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-17T18:07:48.050 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-17T18:07:48.050 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-17T18:07:48.051 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:e255b710:::existing_55:head# 2024-09-17T18:07:48.051 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-17T18:07:48.052 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-17T18:07:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-17T18:07:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:f6809456:::existing_11:head# 2024-09-17T18:07:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-17T18:07:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-17T18:07:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-17T18:07:48.055 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-17T18:07:48.055 INFO:tasks.workunit.client.0.smithi157.stderr:Export successful 2024-09-17T18:07:48.056 INFO:tasks.workunit.client.0.smithi157.stdout: marking collection for removal 2024-09-17T18:07:48.056 INFO:tasks.workunit.client.0.smithi157.stdout:setting '_remove' omap key 2024-09-17T18:07:48.056 INFO:tasks.workunit.client.0.smithi157.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-17T18:07:48.266 INFO:tasks.workunit.client.0.smithi157.stderr:Remove successful 2024-09-17T18:07:48.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33174.out 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-17T18:07:48.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33174.out 2024-09-17T18:07:49.688 INFO:tasks.workunit.client.0.smithi157.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-17T18:07:49.688 INFO:tasks.workunit.client.0.smithi157.stdout:Importing pgid 1.0 2024-09-17T18:07:49.688 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-17T18:07:49.688 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.753 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-17T18:07:49.753 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.786 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-17T18:07:49.786 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.820 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-17T18:07:49.820 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.854 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-17T18:07:49.854 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.887 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-17T18:07:49.887 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.920 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-17T18:07:49.921 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.954 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:169157fa:::existing_81:head# 2024-09-17T18:07:49.954 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:49.987 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:17731382:::existing_58:head# 2024-09-17T18:07:49.987 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.021 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-17T18:07:50.021 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.054 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-17T18:07:50.054 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.088 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-17T18:07:50.088 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.121 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-17T18:07:50.121 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.155 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-17T18:07:50.155 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.188 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:28566edb:::existing_78:head# 2024-09-17T18:07:50.188 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.221 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:2a101182:::existing_20:head# 2024-09-17T18:07:50.222 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.255 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-17T18:07:50.255 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.288 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-17T18:07:50.289 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.322 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-17T18:07:50.322 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.355 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-17T18:07:50.356 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.389 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-17T18:07:50.389 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.422 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-17T18:07:50.422 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.456 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-17T18:07:50.456 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.489 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-17T18:07:50.489 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.522 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-17T18:07:50.523 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.556 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-17T18:07:50.556 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.589 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:53588912:::existing_9:head# 2024-09-17T18:07:50.590 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.623 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-17T18:07:50.623 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.656 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-17T18:07:50.657 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.690 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-17T18:07:50.690 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.723 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-17T18:07:50.724 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.757 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:602f83fe:::foo:head# 2024-09-17T18:07:50.757 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.790 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-17T18:07:50.790 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.823 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:628918ca:::existing_79:head# 2024-09-17T18:07:50.824 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.857 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-17T18:07:50.857 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.890 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-17T18:07:50.891 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.924 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-17T18:07:50.924 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.957 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-17T18:07:50.958 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:50.991 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-17T18:07:50.991 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.024 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-17T18:07:51.025 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.083 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-17T18:07:51.083 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.150 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-17T18:07:51.150 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.200 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:750ace21:::existing_19:head# 2024-09-17T18:07:51.200 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.259 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-17T18:07:51.259 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.309 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:79881152:::existing_63:head# 2024-09-17T18:07:51.309 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.343 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-17T18:07:51.343 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.385 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:81490677:::existing_76:head# 2024-09-17T18:07:51.385 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.426 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-17T18:07:51.427 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.468 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-17T18:07:51.469 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.510 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-17T18:07:51.512 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.552 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-17T18:07:51.552 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.594 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:92731e71:::existing_89:head# 2024-09-17T18:07:51.594 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.635 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:928205cf:::existing_82:head# 2024-09-17T18:07:51.636 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.677 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:93be2267:::existing_24:head# 2024-09-17T18:07:51.677 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.720 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-17T18:07:51.720 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.761 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-17T18:07:51.761 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.803 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-17T18:07:51.803 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.836 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-17T18:07:51.837 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.870 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-17T18:07:51.870 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.903 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a504a533:::existing_90:head# 2024-09-17T18:07:51.903 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.937 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a691337e:::existing_61:head# 2024-09-17T18:07:51.937 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:51.970 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-17T18:07:51.970 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.003 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-17T18:07:52.004 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.037 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-17T18:07:52.037 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.070 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-17T18:07:52.071 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.137 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-17T18:07:52.137 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.171 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:b343e277:::existing_88:head# 2024-09-17T18:07:52.171 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.204 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-17T18:07:52.205 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.238 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-17T18:07:52.238 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.271 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-17T18:07:52.271 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.305 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-17T18:07:52.305 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.338 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:bebec877:::existing_74:head# 2024-09-17T18:07:52.338 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.371 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-17T18:07:52.372 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.405 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:c156033e:::existing_100:head# 2024-09-17T18:07:52.405 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.438 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-17T18:07:52.438 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.472 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-17T18:07:52.472 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.505 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-17T18:07:52.506 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.539 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-17T18:07:52.539 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.572 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-17T18:07:52.572 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.606 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-17T18:07:52.606 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.639 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-17T18:07:52.639 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.673 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-17T18:07:52.673 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.706 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-17T18:07:52.706 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.740 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-17T18:07:52.740 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.773 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-17T18:07:52.773 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.806 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-17T18:07:52.806 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.840 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-17T18:07:52.841 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.874 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-17T18:07:52.874 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.907 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:e255b710:::existing_55:head# 2024-09-17T18:07:52.908 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.941 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-17T18:07:52.941 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:52.974 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-17T18:07:52.974 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.008 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-17T18:07:53.008 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.041 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:f6809456:::existing_11:head# 2024-09-17T18:07:53.041 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.092 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-17T18:07:53.092 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.146 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-17T18:07:53.148 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.217 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-17T18:07:53.217 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.276 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-17T18:07:53.276 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:07:53.326 INFO:tasks.workunit.client.0.smithi157.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-17T18:07:53.368 INFO:tasks.workunit.client.0.smithi157.stdout:Import successful 2024-09-17T18:07:53.888 INFO:tasks.workunit.client.0.smithi157.stdout:reviving divergent 1 2024-09-17T18:07:53.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-17T18:07:53.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:07:54.269 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:07:54.270 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-17T18:06:22.986217+0000 41'101 43:137 [0,2] 0 [0,2] 0 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:33:58.218308+0000 0 0 2024-09-17T18:07:54.270 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:07:54.270 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:07:54.270 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:07:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-17T18:07:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:07:54.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:07:54.284 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:07:54.285 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:07:54.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:07:54.286 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:07:54.286 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:07:54.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:07:54.318 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:54.317+0000 7f6439145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:54.320 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:54.319+0000 7f6439145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:54.321 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:54.320+0000 7f6439145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:07:54.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:07:54.677 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:07:54.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:07:54.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:07:55.123 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/235845353,v1:127.0.0.1:6819/235845353] [v2:127.0.0.1:6820/235845353,v1:127.0.0.1:6821/235845353] exists,up fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:07:55.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:07:55.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:07:55.125 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:07:55.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:07:55.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:07:55.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:07:55.400 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:55.399+0000 7f6439145640 -1 Falling back to public interface 2024-09-17T18:07:55.584 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/235845353,v1:127.0.0.1:6819/235845353] [v2:127.0.0.1:6820/235845353,v1:127.0.0.1:6821/235845353] exists,up fa7bc93c-a136-4a32-920f-60afc691be80 2024-09-17T18:07:55.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:07:55.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:07:55.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:07:55.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-17T18:07:58.359 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:07:58.358+0000 7f6439145640 -1 osd.1 47 log_to_monitors true 2024-09-17T18:08:15.587 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-17T18:08:15.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T18:08:15.675 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:08:15.675 INFO:tasks.workunit.client.0.smithi157.stdout: "ops": [], 2024-09-17T18:08:15.675 INFO:tasks.workunit.client.0.smithi157.stdout: "num_ops": 0 2024-09-17T18:08:15.675 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:08:15.682 INFO:tasks.workunit.client.0.smithi157.stdout:allowing recovery 2024-09-17T18:08:15.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-17T18:08:15.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:08:16.056 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:08:16.056 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-17T18:07:59.960110+0000 47'102 51:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:43:22.450869+0000 0 0 2024-09-17T18:08:16.056 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:08:16.056 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:08:16.056 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:08:16.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-17T18:08:16.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-17T18:08:16.190 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:08:16.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-17T18:08:16.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-17T18:08:16.312 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:08:16.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-17T18:08:16.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-17T18:08:16.434 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:08:16.443 INFO:tasks.workunit.client.0.smithi157.stdout:reading divergent objects 2024-09-17T18:08:16.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-17T18:08:16.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-17T18:08:16.815 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:08:16.815 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-17T18:07:59.960110+0000 47'102 51:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:05:24.376092+0000 0'0 2024-09-17T18:05:24.376092+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:43:22.450869+0000 0 0 2024-09-17T18:08:16.815 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:08:16.815 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:08:16.815 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:08:16.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-17T18:08:16.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-17T18:08:16.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:08:16.828 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-17T18:08:16.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:08:16.864 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-17T18:08:16.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:08:16.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-17T18:08:16.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:08:16.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-17T18:08:16.972 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-17T18:08:16.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:08:16.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-17T18:08:17.006 INFO:tasks.workunit.client.0.smithi157.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-17T18:08:17.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:08:17.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-17T18:08:17.054 INFO:tasks.workunit.client.0.smithi157.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-17T18:08:17.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-17T18:08:17.057 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-17T18:08:17.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:08:17.062 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4638.0:1 2024-09-17T18:05:57.534347+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:08:17.062 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'105 (29'4) delete 1:70b3bf12:::existing_4:head by client.4638.0:1 2024-09-17T18:05:57.534347+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:08:17.062 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:08:17.062 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-17T18:08:17.062 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-09-17T18:07:00.046306+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-17T18:05:57.528669+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-17T18:05:57.528669+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-17T18:05:57.534799+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-17T18:05:57.534799+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:08:17.063 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-09-17T18:05:57.534799+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-17T18:05:57.526145+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-17T18:05:57.526145+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-17T18:05:57.526145+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-17T18:08:17.064 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-17T18:05:57.507406+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-17T18:05:57.507406+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-17T18:08:17.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-17T18:05:57.507406+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4633.0:1 2024-09-17T18:05:57.528822+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 20 _merge_object_divergent_entries: keeping 29'104 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4633.0:1 2024-09-17T18:05:57.528822+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:07:45.053+0000 7febf8b7d640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-17T18:08:17.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-17T18:08:17.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:08:17.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-17T18:08:17.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:08:17.075 INFO:tasks.workunit.client.0.smithi157.stdout:success 2024-09-17T18:08:17.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-17T18:08:17.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.33174.out 2024-09-17T18:08:17.076 INFO:tasks.workunit.client.0.smithi157.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33174.out': No such file or directory 2024-09-17T18:08:17.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-17T18:08:17.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-17T18:08:17.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-17T18:08:17.559 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' does not exist 2024-09-17T18:08:17.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-17T18:08:17.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:08:17.572 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:08:17.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:08:17.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:08:17.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:08:22.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:08:22.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:08:22.892 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:08:22.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:08:22.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:08:22.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:08:22.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:08:22.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:08:22.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:08:22.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:08:22.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:08:22.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:08:22.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:08:22.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:08:22.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:08:22.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:08:22.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:22.923 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:22.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-17T18:08:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:08:22.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:08:22.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:08:22.927 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:08:22.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:08:22.928 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:08:22.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:08:22.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:08:22.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:08:22.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:08:22.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:08:22.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:08:22.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:08:22.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:08:22.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:08:22.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:08:22.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:08:22.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:22.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:22.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:08:22.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:08:22.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:08:22.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-17T18:08:22.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:08:22.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:22.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:22.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33174 2024-09-17T18:08:22.939 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:08:22.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-17T18:08:22.940 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-17T18:08:22.941 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:08:22.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-17T18:08:22.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:23.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:08:23.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:08:23.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:08:23.614 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:08:23.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:08:23.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:08:23.615 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:08:23.616 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get fsid 2024-09-17T18:08:23.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:08:23.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:08:23.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:08:23.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:08:23.701 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:08:23.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get mon_host 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-17T18:08:23.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:08:24.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:24.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:08:24.638 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:08:24.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-17T18:08:24.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-17T18:08:24.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:08:24.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:08:24.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:08:24.674 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:08:24.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:08:24.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bfecdf44-7455-4b04-8a35-f511ef8dd599' 2024-09-17T18:08:24.677 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:08:24.677 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:08:24.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYxelmVviCKRAAYq2WMNioOOw5Zbf5cqGUWA== 2024-09-17T18:08:24.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYxelmVviCKRAAYq2WMNioOOw5Zbf5cqGUWA=="}' 2024-09-17T18:08:24.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bfecdf44-7455-4b04-8a35-f511ef8dd599 -i td/divergent-priors/0/new.json 2024-09-17T18:08:24.963 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:08:24.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-17T18:08:24.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYxelmVviCKRAAYq2WMNioOOw5Zbf5cqGUWA== --osd-uuid bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:08:25.012 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:25.010+0000 7fb3ece2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:25.023 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:25.021+0000 7fb3ece2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:25.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:25.022+0000 7fb3ece2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:25.024 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:25.022+0000 7fb3ece2a640 -1 bdev(0x55d655903c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:08:25.025 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:25.022+0000 7fb3ece2a640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-17T18:08:30.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-17T18:08:30.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:08:30.522 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:08:30.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:08:30.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:08:31.074 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:08:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:08:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:08:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:08:31.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:08:31.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:08:31.110 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:31.107+0000 7fafb4f41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:31.111 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:31.109+0000 7fafb4f41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:31.113 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:31.110+0000 7fafb4f41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:31.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:08:31.517 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:08:31.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:31.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:31.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:32.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:32.227+0000 7fafb4f41640 -1 Falling back to public interface 2024-09-17T18:08:32.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:32.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:32.907 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:08:32.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:08:32.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:32.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:33.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:34.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:34.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:34.345 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:08:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:08:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:34.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:35.380 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:35.378+0000 7fafb4f41640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:08:35.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:35.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:35.744 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:08:35.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:08:35.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:35.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:36.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:37.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:37.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:37.138 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:08:37.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:08:37.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:37.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:37.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:38.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:38.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:38.573 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:08:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:08:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:38.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:39.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:40.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:40.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:40.127 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:08:40.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:08:40.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:40.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:40.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:41.538 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:08:41.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:08:41.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:41.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:41.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:42.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:42.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:42.943 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:08:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:08:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:42.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:43.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:44.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:44.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:44.432 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:08:44.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:08:44.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:44.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:44.867 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:45.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:45.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:45.869 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:08:45.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:08:45.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:45.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:46.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:47.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:47.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:47.296 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:08:47.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:08:47.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:47.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:48.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:49.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:49.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:49.046 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:08:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:08:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:49.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:49.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:50.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:50.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:50.913 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:08:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:08:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:50.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:51.975 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:52.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:52.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:52.977 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:08:52.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:08:52.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:52.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:53.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:54.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:54.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:54.533 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:08:54.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:08:54.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:54.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:54.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:55.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:55.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:55.988 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:08:55.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:08:55.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:55.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:56.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:08:57.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:08:57.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:08:57.764 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:08:57.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:08:57.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:08:57.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:08:58.166 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4095238311,v1:127.0.0.1:6803/4095238311] [v2:127.0.0.1:6804/4095238311,v1:127.0.0.1:6805/4095238311] exists,up bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:08:58.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:08:58.171 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:08:58.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 abefbbb4-172f-4bc4-b393-5e13b15cc9d5' 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:08:58.173 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:08:58.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6xelmeQRxCxAAZMQe33veXg6fhJ3uMYmEjA== 2024-09-17T18:08:58.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6xelmeQRxCxAAZMQe33veXg6fhJ3uMYmEjA=="}' 2024-09-17T18:08:58.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new abefbbb4-172f-4bc4-b393-5e13b15cc9d5 -i td/divergent-priors/1/new.json 2024-09-17T18:08:58.794 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:08:58.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-17T18:08:58.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6xelmeQRxCxAAZMQe33veXg6fhJ3uMYmEjA== --osd-uuid abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:08:58.844 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:58.843+0000 7ff4a4a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:58.869 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:58.867+0000 7ff4a4a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:58.870 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:58.868+0000 7ff4a4a30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:08:58.870 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:58.869+0000 7ff4a4a30640 -1 bdev(0x55b6cfa0bc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:08:58.870 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:08:58.869+0000 7ff4a4a30640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-17T18:09:04.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-17T18:09:04.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:09:04.541 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:09:04.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:09:04.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:09:05.087 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:09:05.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:09:05.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:09:05.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:09:05.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:09:05.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:09:05.122 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:05.121+0000 7f7c80418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:05.124 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:05.123+0000 7f7c80418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:05.125 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:05.124+0000 7f7c80418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:05.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:09:05.555 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:09:05.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:05.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:05.966 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:05.964+0000 7f7c80418640 -1 Falling back to public interface 2024-09-17T18:09:06.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:07.089 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:09:07.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:09:07.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:07.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:07.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:08.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:08.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:08.524 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:09:08.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:09:08.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:08.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:09.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:09.319 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:09.316+0000 7f7c80418640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:09:10.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:10.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:10.002 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:09:10.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:09:10.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:10.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:10.480 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:11.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:11.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:11.482 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:09:11.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:09:11.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:11.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:11.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:12.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:12.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:12.900 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:09:12.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:09:12.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:12.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:13.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:14.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:14.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:14.303 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:09:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:09:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:14.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:15.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:15.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:15.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:09:15.749 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:09:15.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:15.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:16.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:17.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:17.272 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:09:17.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:09:17.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:17.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:17.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:18.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:18.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:18.694 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:09:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:09:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:18.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:19.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:20.202 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:09:20.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:09:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:20.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:20.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:21.606 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:09:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:09:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:22.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:23.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:23.009 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:23.009 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:09:23.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:09:23.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:23.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:24.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:25.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:25.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:25.007 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:09:25.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:09:25.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:25.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:25.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:26.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:26.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:26.701 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:09:26.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:09:26.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:26.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:27.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:28.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:28.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:28.619 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:09:28.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:09:28.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:28.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:29.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:30.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:30.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:30.515 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:09:30.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:09:30.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:30.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:31.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:32.138 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:09:32.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:32.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:32.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:09:32.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:32.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:09:32.637 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1332764119,v1:127.0.0.1:6811/1332764119] [v2:127.0.0.1:6812/1332764119,v1:127.0.0.1:6813/1332764119] exists,up abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:09:32.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:09:32.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:09:32.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 dd3049bf-4a8b-4a5d-a1cc-256f71296e70' 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:09:32.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:09:32.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDcxelmXFOIJxAAW7FC24t8qjPbf9d8NN0S0w== 2024-09-17T18:09:32.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDcxelmXFOIJxAAW7FC24t8qjPbf9d8NN0S0w=="}' 2024-09-17T18:09:32.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dd3049bf-4a8b-4a5d-a1cc-256f71296e70 -i td/divergent-priors/2/new.json 2024-09-17T18:09:33.170 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:09:33.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-17T18:09:33.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcxelmXFOIJxAAW7FC24t8qjPbf9d8NN0S0w== --osd-uuid dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:09:33.218 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:33.216+0000 7f623dc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:33.223 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:33.220+0000 7f623dc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:33.224 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:33.221+0000 7f623dc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:33.224 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:33.222+0000 7f623dc2c640 -1 bdev(0x559a531f5c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:09:33.224 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:33.222+0000 7f623dc2c640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-17T18:09:39.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-17T18:09:39.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:09:39.473 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:09:39.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:09:39.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:09:40.013 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:09:40.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:09:40.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:09:40.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:09:40.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:09:40.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:09:40.049 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:40.047+0000 7f0c1ab45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:40.056 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:40.054+0000 7f0c1ab45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:40.057 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:40.055+0000 7f0c1ab45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:09:40.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:09:40.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:09:40.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:09:40.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:09:40.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:09:40.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:40.446 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:09:40.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:09:40.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:40.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:40.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:41.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:41.646+0000 7f0c1ab45640 -1 Falling back to public interface 2024-09-17T18:09:41.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:41.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:41.881 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:09:41.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:09:41.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:41.881 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:42.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:43.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:43.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:43.316 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:09:43.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:09:43.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:43.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:43.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:44.219 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:09:44.217+0000 7f0c1ab45640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:09:44.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:44.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:44.720 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:09:44.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:09:44.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:44.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:45.349 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:46.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:46.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:46.351 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:09:46.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:09:46.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:46.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:46.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:47.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:47.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:47.778 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:09:47.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:09:47.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:47.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:48.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:49.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:49.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:49.245 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:09:49.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:09:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:49.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:50.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:50.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:50.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:09:50.684 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:09:50.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:50.684 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:51.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:52.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:52.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:52.108 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:09:52.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:09:52.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:52.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:52.534 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:53.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:53.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:53.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:09:53.536 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:09:53.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:53.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:54.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:55.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:55.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:55.074 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:09:55.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:09:55.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:55.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:55.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:56.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:56.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:56.624 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:09:56.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:09:56.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:56.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:57.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:09:58.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:09:58.352 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:09:58.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:09:58.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:09:58.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:09:58.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:09:59.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:10:00.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:10:00.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:10:00.123 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:10:00.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:10:00.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:10:00.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:10:00.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:10:01.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:10:01.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:10:01.626 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:10:01.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:10:01.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:10:01.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:10:02.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:10:03.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:10:03.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:10:03.236 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:10:03.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:10:03.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:10:03.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:10:04.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:10:05.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:10:05.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:10:05.300 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:10:05.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:10:05.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:10:05.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:10:06.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:10:07.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:10:07.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:10:07.052 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:10:07.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:10:07.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:10:07.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:10:07.455 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1681996688,v1:127.0.0.1:6819/1681996688] [v2:127.0.0.1:6820/1681996688,v1:127.0.0.1:6821/1681996688] exists,up dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:10:07.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:10:07.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:10:07.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:10:07.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-17T18:10:08.061 INFO:tasks.workunit.client.0.smithi157.stderr:noout is set 2024-09-17T18:10:08.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-17T18:10:08.957 INFO:tasks.workunit.client.0.smithi157.stderr:noin is set 2024-09-17T18:10:08.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-17T18:10:09.779 INFO:tasks.workunit.client.0.smithi157.stderr:nodown is set 2024-09-17T18:10:09.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-17T18:10:09.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-17T18:10:10.389 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-17T18:10:10.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:10:11.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-17T18:10:12.037 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 size to 3 2024-09-17T18:10:12.049 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-17T18:10:13.002 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 min_size to 2 2024-09-17T18:10:13.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-17T18:10:13.822 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 pg_autoscale_mode to on 2024-09-17T18:10:13.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-17T18:10:13.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726596613 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:10:13.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:10:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:10:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:10:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:10:14.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:10:14.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:14.305 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:10:14.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-17T18:10:14.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-17T18:10:14.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-17T18:10:14.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:14.425 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:10:14.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T18:10:14.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T18:10:14.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977' 2024-09-17T18:10:14.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:14.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:10:14.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-17T18:10:14.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-17T18:10:14.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672977 2-64424509446' 2024-09-17T18:10:14.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:14.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-17T18:10:14.667 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:14.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:10:14.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-17T18:10:14.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:14.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-17T18:10:14.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-17T18:10:14.669 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836504 2024-09-17T18:10:14.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:10:15.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-17T18:10:15.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:15.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T18:10:15.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:15.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:10:15.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T18:10:15.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:15.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T18:10:15.110 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672977 2024-09-17T18:10:15.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T18:10:15.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:10:15.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-17T18:10:15.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:15.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-17T18:10:15.575 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:15.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:10:15.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-17T18:10:15.576 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:15.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-17T18:10:15.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-17T18:10:15.578 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509446 2024-09-17T18:10:15.578 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:10:15.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-17T18:10:15.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-17T18:10:15.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:10:15.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:10:15.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:10:15.994 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:10:15.994 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:10:15.995 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:10:15.995 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:10:15.995 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:10:15.995 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:10:16.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:16.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:10:16.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-17T18:10:16.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-17T18:10:16.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-17T18:10:16.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:16.641 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:10:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-17T18:10:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-17T18:10:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-09-17T18:10:16.764 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:16.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:10:16.912 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509448' 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:16.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:10:16.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-17T18:10:16.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:16.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-17T18:10:16.915 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836506 2024-09-17T18:10:16.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-17T18:10:16.916 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:10:17.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-17T18:10:17.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:17.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-17T18:10:17.347 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:17.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:10:17.349 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-17T18:10:17.349 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:17.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-17T18:10:17.350 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672979 2024-09-17T18:10:17.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-17T18:10:17.350 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:10:17.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672979 2024-09-17T18:10:17.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:10:18.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:10:18.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:10:19.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-17T18:10:19.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:19.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-17T18:10:19.226 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:19.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:10:19.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-17T18:10:19.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:19.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-17T18:10:19.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-17T18:10:19.229 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509448 2024-09-17T18:10:19.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:10:19.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-17T18:10:19.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:10:19.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:10:19.702 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:10:20.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:10:20.633 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:10:20.633 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:10:20.633 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:10:20.634 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:10:21.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:10:21.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:10:21.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:10:21.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-17T18:10:21.170 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-17T18:10:21.561 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:10:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-17T18:10:21.573 INFO:tasks.workunit.client.0.smithi157.stdout:primary and soon to be divergent is 1 2024-09-17T18:10:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-17T18:10:21.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:10:21.951 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:10:21.951 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:10:12.381065+0000 0'0 29:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:56:24.432415+0000 0 0 2024-09-17T18:10:21.951 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:10:21.951 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:10:21.952 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:10:21.962 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-17T18:10:21.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726596621 2024-09-17T18:10:21.963 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726596621 - 1726596613 2024-09-17T18:10:21.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=8 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 8 -gt 300 ']' 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-17T18:10:21.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-17T18:10:21.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-17T18:10:21.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-17T18:10:21.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-17T18:10:21.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-17T18:10:21.967 INFO:tasks.workunit.client.0.smithi157.stdout:writing initial objects 2024-09-17T18:10:21.968 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-17T18:10:21.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:21.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-17T18:10:22.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-17T18:10:22.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-17T18:10:22.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.406 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-17T18:10:22.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-17T18:10:22.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-17T18:10:22.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:22.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-17T18:10:23.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-17T18:10:23.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-17T18:10:23.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-17T18:10:23.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-17T18:10:23.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-17T18:10:23.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-17T18:10:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:23.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-17T18:10:24.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-17T18:10:24.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-17T18:10:24.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-17T18:10:24.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-17T18:10:24.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-17T18:10:24.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.763 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-17T18:10:24.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:24.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-17T18:10:25.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-17T18:10:25.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-17T18:10:25.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-17T18:10:25.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-17T18:10:25.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-17T18:10:25.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-17T18:10:25.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:25.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-17T18:10:26.129 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-17T18:10:26.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-17T18:10:26.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-17T18:10:26.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.565 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-17T18:10:26.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-17T18:10:26.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:26.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-17T18:10:27.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-17T18:10:27.185 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-17T18:10:27.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-17T18:10:27.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-17T18:10:27.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-17T18:10:27.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-17T18:10:27.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:27.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-17T18:10:28.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-17T18:10:28.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-17T18:10:28.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-17T18:10:28.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-17T18:10:28.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-17T18:10:28.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-17T18:10:28.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:28.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-17T18:10:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-17T18:10:29.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-17T18:10:29.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-17T18:10:29.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-17T18:10:29.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-17T18:10:29.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:29.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-17T18:10:30.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-17T18:10:30.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-17T18:10:30.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-17T18:10:30.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.555 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-17T18:10:30.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-17T18:10:30.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:30.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-17T18:10:32.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:32.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-17T18:10:32.282 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:32.283 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-17T18:10:32.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:32.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-17T18:10:32.710 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:32.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-17T18:10:32.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:32.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-17T18:10:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-17T18:10:33.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-17T18:10:33.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-17T18:10:33.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-17T18:10:33.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.709 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-17T18:10:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:33.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-17T18:10:34.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-17T18:10:34.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-17T18:10:34.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-17T18:10:34.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-17T18:10:34.698 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.699 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-17T18:10:34.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:34.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-17T18:10:35.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:35.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-17T18:10:35.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:35.580 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-17T18:10:35.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:35.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-17T18:10:36.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:36.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-17T18:10:36.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:36.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-17T18:10:36.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:36.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-17T18:10:36.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:36.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-17T18:10:36.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:36.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-17T18:10:37.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:37.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-17T18:10:37.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:37.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-17T18:10:38.640 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:38.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-17T18:10:38.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:38.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-17T18:10:39.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:39.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-17T18:10:39.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:39.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-17T18:10:40.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-17T18:10:41.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:41.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-17T18:10:41.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:41.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-17T18:10:41.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:41.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-17T18:10:41.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:41.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-17T18:10:41.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:41.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-17T18:10:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:42.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-17T18:10:42.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:42.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-17T18:10:42.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-17T18:10:42.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:10:42.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:10:42.782 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:10:42.782 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:10:42.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:10:42.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:10:42.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:10:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:10:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:10:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:10:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:10:42.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:10:42.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:10:43.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:10:43.317 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:10:43.318 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:10:43.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:10:43.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:43.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:10:43.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-17T18:10:43.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-17T18:10:43.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-17T18:10:43.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:43.468 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:10:43.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-17T18:10:43.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-17T18:10:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672987' 2024-09-17T18:10:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:10:43.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672987 2-64424509456' 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:43.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:10:43.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-17T18:10:43.741 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:43.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-17T18:10:43.742 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836513 2024-09-17T18:10:43.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-17T18:10:43.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:10:44.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-17T18:10:44.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:10:45.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:10:45.139 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:10:45.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836513 2024-09-17T18:10:45.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:45.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-17T18:10:45.781 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:45.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:10:45.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-17T18:10:45.782 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:45.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-17T18:10:45.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-17T18:10:45.783 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672987 2024-09-17T18:10:45.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:10:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-17T18:10:46.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:10:46.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-17T18:10:46.198 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:10:46.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:10:46.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-17T18:10:46.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:10:46.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-17T18:10:46.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-17T18:10:46.201 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509456 2024-09-17T18:10:46.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:10:46.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-17T18:10:46.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:10:46.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:10:46.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:10:47.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:10:47.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:10:47.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-17T18:10:47.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:10:47.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:10:47.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:10:48.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-17T18:10:48.149 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:10:48.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-17T18:10:48.686 INFO:tasks.workunit.client.0.smithi157.stdout:blackholing osds 0 2 2024-09-17T18:10:48.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-17T18:10:48.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:10:49.061 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:10:49.061 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:42.171035+0000 32'60 43:131 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:20:36.763403+0000 3 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:45.951519+0000 39'61 43:138 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:11:19.474604+0000 2 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:10:38.906526+0000 32'61 43:142 [2,0,1] 2 [2,0,1] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:44:55.486298+0000 4 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:46.112214+0000 38'60 43:131 [1,2,0] 1 [1,2,0] 1 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:35:09.892339+0000 3 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:10:41.835611+0000 30'59 43:123 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:14:47.144289+0000 3 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:10:43.909512+0000 36'61 43:130 [2,0,1] 2 [2,0,1] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:44:20.534977+0000 5 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 0 0 0 1857 0 0 19 42 19 active+clean 2024-09-17T18:10:31.940821+0000 38'61 43:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.062 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 891 0 0 17 42 17 active+clean 2024-09-17T18:10:40.399712+0000 30'59 43:122 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.063 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 5 0 0 0 0 2340 0 0 20 42 20 active+clean 2024-09-17T18:10:45.850797+0000 34'62 43:119 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.063 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 active+clean 2024-09-17T18:10:41.013675+0000 36'62 43:143 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:49:24.360889+0000 6 0 2024-09-17T18:10:49.063 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:34.293514+0000 30'59 43:134 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:56:52.902339+0000 2 0 2024-09-17T18:10:49.063 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:10:40.518636+0000 39'61 43:132 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:44:39.216740+0000 4 0 2024-09-17T18:10:49.063 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 active+clean 2024-09-17T18:10:44.804030+0000 32'60 43:136 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:01:01.341433+0000 1 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:41.139586+0000 35'60 43:135 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:11:34.799123+0000 3 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 active+clean 2024-09-17T18:10:44.938812+0000 34'61 43:141 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:46:02.029199+0000 2 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:38.249909+0000 30'59 43:134 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:02:37.118104+0000 2 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:32.273255+0000 38'60 43:135 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:11:55.440348+0000 2 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:41.995132+0000 36'61 43:130 [2,0,1] 2 [2,0,1] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T04:49:21.419518+0000 3 0 2024-09-17T18:10:49.064 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 0 0 0 1857 0 0 19 42 19 active+clean 2024-09-17T18:10:45.850609+0000 36'61 43:126 [1,0,2] 1 [1,0,2] 1 36'61 2024-09-17T18:10:42.778098+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 0 0 0 1374 0 0 18 42 18 active+clean 2024-09-17T18:10:40.517718+0000 32'60 43:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:45.154561+0000 39'60 43:135 [1,0,2] 1 [1,0,2] 1 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:16:12.202318+0000 3 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:10:36.883182+0000 30'59 43:128 [2,0,1] 2 [2,0,1] 2 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:34:31.322951+0000 3 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:10:38.797865+0000 30'59 43:123 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:13:26.462798+0000 1 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:44.076658+0000 39'61 43:132 [1,2,0] 1 [1,2,0] 1 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:54:31.924534+0000 3 0 2024-09-17T18:10:49.065 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:40.157462+0000 39'61 43:136 [1,0,2] 1 [1,0,2] 1 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:32:28.108262+0000 2 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:42.935525+0000 32'60 43:141 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T04:43:01.802210+0000 3 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:10:47.151112+0000 32'61 43:129 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:35:44.966709+0000 4 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 0 0 0 890 0 0 17 42 17 active+clean 2024-09-17T18:10:40.519010+0000 30'59 43:122 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:39.284337+0000 39'61 43:136 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:01:25.515938+0000 2 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:39.938583+0000 30'59 43:128 [2,0,1] 2 [2,0,1] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:34:18.341895+0000 2 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:10:38.311824+0000 34'60 43:141 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:30:11.537204+0000 3 0 2024-09-17T18:10:49.066 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:10:43.099787+0000 39'61 43:132 [1,2,0] 1 [1,2,0] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:37:40.799679+0000 5 0 2024-09-17T18:10:49.067 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:10:49.067 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:10:49.067 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:10:49.072 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:10:49.073 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:10:49.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-17T18:10:49.153 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:10:49.153 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:10:49.153 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:10:49.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:10:49.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-17T18:10:49.239 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:10:49.239 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:10:49.239 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:10:49.246 INFO:tasks.workunit.client.0.smithi157.stdout:create new divergent objects 2024-09-17T18:10:49.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-17T18:10:49.247 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-17T18:10:49.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:10:49.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:10:49.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-17T18:10:49.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-17T18:10:49.248 INFO:tasks.workunit.client.0.smithi157.stdout:writing divergent objects 2024-09-17T18:10:49.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-17T18:10:49.249 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-17T18:10:49.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-17T18:10:49.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-17T18:10:49.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-17T18:10:49.250 INFO:tasks.workunit.client.0.smithi157.stdout:remove divergent objects 2024-09-17T18:10:49.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-17T18:10:49.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-17T18:10:49.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-17T18:10:49.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-17T18:10:49.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-17T18:10:49.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-17T18:10:49.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-17T18:10:49.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-17T18:10:49.253 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-17T18:10:49.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-17T18:10:49.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-17T18:10:49.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-17T18:10:59.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-17T18:10:59.260 INFO:tasks.workunit.client.0.smithi157.stderr:rados: no process found 2024-09-17T18:10:59.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-17T18:10:59.261 INFO:tasks.workunit.client.0.smithi157.stdout:killing all the osds 2024-09-17T18:10:59.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:42.171035+0000 32'60 52:154 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:39:52.643115+0000 3 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:45.951519+0000 39'61 52:161 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:08:35.264444+0000 2 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:10:38.906526+0000 32'61 52:165 [2,0,1] 2 [2,0,1] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:46:21.633718+0000 4 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:55.395824+0000 38'60 52:160 [1,0,2] 1 [1,0,2] 1 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:22.768293+0000 3 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:10:55.247322+0000 30'59 52:129 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:55.247230+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:57:00.740224+0000 3 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 0 0 0 2086 0 0 20 42 20 active+clean 2024-09-17T18:10:52.219505+0000 45'62 52:167 [1,2,0] 1 [1,2,0] 1 45'62 2024-09-17T18:10:52.219421+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:05:58.335004+0000 5 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:53.240477+0000 38'61 52:165 [1,2,0] 1 [1,2,0] 1 38'61 2024-09-17T18:10:53.240404+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:25:23.071432+0000 3 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:51.192518+0000 30'59 52:153 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:51.192459+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:25:29.739288+0000 2 0 2024-09-17T18:10:59.706 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 0 0 0 1932 0 0 20 42 20 active+clean 2024-09-17T18:10:56.202924+0000 34'62 52:150 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:56.202865+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:15:36.632994+0000 4 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 active+clean 2024-09-17T18:10:41.013675+0000 36'62 52:166 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:23.701200+0000 6 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:34.293514+0000 30'59 52:157 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:53:35.735851+0000 2 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:10:40.518636+0000 39'61 52:155 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:44:24.214360+0000 4 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 active+clean 2024-09-17T18:10:44.804030+0000 32'60 52:159 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:47:16.111898+0000 1 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:41.139586+0000 35'60 52:158 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:26:01.279125+0000 3 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 active+clean 2024-09-17T18:10:44.938812+0000 34'61 52:164 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:51:47.313653+0000 2 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:38.249909+0000 30'59 52:157 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:53:23.130889+0000 2 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:32.273255+0000 38'60 52:158 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:05:38.868271+0000 2 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:41.995132+0000 36'61 52:153 [2,0,1] 2 [2,0,1] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:57:43.085917+0000 3 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 0 0 0 1449 0 0 21 42 21 active+clean 2024-09-17T18:10:48.198874+0000 45'63 52:159 [1,0,2] 1 [1,0,2] 1 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T04:31:38.310473+0000 4 0 2024-09-17T18:10:59.707 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:10:50.212261+0000 32'60 52:154 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:50.212200+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:14:58.622073+0000 4 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:45.154561+0000 39'60 52:158 [1,0,2] 1 [1,0,2] 1 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:10:38.607167+0000 3 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 0 0 0 966 0 0 18 42 18 active+clean 2024-09-17T18:10:36.883182+0000 45'60 52:152 [2,0,1] 2 [2,0,1] 2 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:17:22.200363+0000 3 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:10:38.797865+0000 30'59 52:146 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:25:35.004186+0000 1 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:44.076658+0000 39'61 52:155 [1,2,0] 1 [1,2,0] 1 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:06:01.867684+0000 3 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:40.157462+0000 39'61 52:159 [1,0,2] 1 [1,0,2] 1 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:18:56.749701+0000 2 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:10:42.935525+0000 32'60 52:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:25:19.215561+0000 3 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:10:47.151112+0000 32'61 52:152 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:38:16.562950+0000 4 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:10:49.259656+0000 30'59 52:153 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:49.259560+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:35:42.898984+0000 1 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:10:49.535194+0000 39'61 52:165 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:08:26.760792+0000 2 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:10:54.238888+0000 30'59 52:129 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:54.238827+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:12:49.484540+0000 2 0 2024-09-17T18:10:59.708 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:10:53.382926+0000 34'60 52:130 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:05:39.760025+0000 3 0 2024-09-17T18:10:59.709 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 6 0 0 0 0 2086 0 0 21 42 21 active+clean 2024-09-17T18:10:55.395862+0000 45'63 52:163 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:19:53.639869+0000 5 0 2024-09-17T18:10:59.709 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:10:59.709 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:10:59.709 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:10:59.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-17T18:10:59.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:10:59.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:10:59.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:10:59.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:10:59.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:10:59.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:10:59.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-17T18:10:59.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-17T18:11:00.325 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:11:00.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-17T18:11:00.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-17T18:11:00.968 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:11:00.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-17T18:11:00.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-17T18:11:01.526 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:11:01.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:01.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-17T18:11:02.045 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already out. 2024-09-17T18:11:02.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:02.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-17T18:11:02.545 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already out. 2024-09-17T18:11:02.556 INFO:tasks.workunit.client.0.smithi157.stdout:bringing up non_divergent 0 2 2024-09-17T18:11:02.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-17T18:11:02.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:11:02.930 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:11:02.930 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:42.171035+0000 32'60 52:154 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:39:52.643115+0000 3 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:45.951519+0000 39'61 52:161 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:08:35.264444+0000 2 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:38.906526+0000 32'61 52:165 [2,0,1] 2 [2,0,1] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:46:21.633718+0000 4 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:55.395824+0000 38'60 52:160 [1,0,2] 1 [1,0,2] 1 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:22.768293+0000 3 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:55.247322+0000 30'59 52:129 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:55.247230+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:57:00.740224+0000 3 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 0 0 0 2086 0 0 20 42 20 stale+active+clean 2024-09-17T18:10:52.219505+0000 45'62 52:167 [1,2,0] 1 [1,2,0] 1 45'62 2024-09-17T18:10:52.219421+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:05:58.335004+0000 5 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:53.240477+0000 38'61 52:165 [1,2,0] 1 [1,2,0] 1 38'61 2024-09-17T18:10:53.240404+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:25:23.071432+0000 3 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 966 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:51.192518+0000 30'59 52:153 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:51.192459+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:25:29.739288+0000 2 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 0 0 0 1932 0 0 20 42 20 stale+active+clean 2024-09-17T18:10:56.202924+0000 34'62 52:150 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:56.202865+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:15:36.632994+0000 4 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 stale+active+clean 2024-09-17T18:10:41.013675+0000 36'62 52:166 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:23.701200+0000 6 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 966 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:34.293514+0000 30'59 52:157 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:53:35.735851+0000 2 0 2024-09-17T18:11:02.931 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:40.518636+0000 39'61 52:155 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:44:24.214360+0000 4 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:44.804030+0000 32'60 52:159 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:47:16.111898+0000 1 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:41.139586+0000 35'60 52:158 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:26:01.279125+0000 3 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:44.938812+0000 34'61 52:164 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:51:47.313653+0000 2 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 2 0 0 0 0 966 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:38.249909+0000 30'59 52:157 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:53:23.130889+0000 2 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:32.273255+0000 38'60 52:158 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:05:38.868271+0000 2 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:41.995132+0000 36'61 52:153 [2,0,1] 2 [2,0,1] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:57:43.085917+0000 3 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 0 0 0 1449 0 0 21 42 21 stale+active+clean 2024-09-17T18:10:48.198874+0000 45'63 52:159 [1,0,2] 1 [1,0,2] 1 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T04:31:38.310473+0000 4 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 4 0 0 0 0 1932 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:50.212261+0000 32'60 52:154 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:50.212200+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:14:58.622073+0000 4 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:45.154561+0000 39'60 52:158 [1,0,2] 1 [1,0,2] 1 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:10:38.607167+0000 3 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 0 0 0 966 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:36.883182+0000 45'60 52:152 [2,0,1] 2 [2,0,1] 2 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:17:22.200363+0000 3 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:38.797865+0000 30'59 52:146 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:25:35.004186+0000 1 0 2024-09-17T18:11:02.932 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:44.076658+0000 39'61 52:155 [1,2,0] 1 [1,2,0] 1 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:06:01.867684+0000 3 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:40.157462+0000 39'61 52:159 [1,0,2] 1 [1,0,2] 1 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:18:56.749701+0000 2 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:42.935525+0000 32'60 52:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:25:19.215561+0000 3 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:47.151112+0000 32'61 52:152 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:38:16.562950+0000 4 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 1 0 0 0 0 483 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:49.259656+0000 30'59 52:153 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:49.259560+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:35:42.898984+0000 1 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 stale+active+clean 2024-09-17T18:10:49.535194+0000 39'61 52:165 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:08:26.760792+0000 2 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 966 0 0 17 42 17 stale+active+clean 2024-09-17T18:10:54.238888+0000 30'59 52:129 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:54.238827+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:12:49.484540+0000 2 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 stale+active+clean 2024-09-17T18:10:53.382926+0000 34'60 52:130 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:05:39.760025+0000 3 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 6 0 0 0 0 2086 0 0 21 42 21 stale+active+clean 2024-09-17T18:10:55.395862+0000 45'63 52:163 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:19:53.639869+0000 5 0 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:11:02.933 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:11:02.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:11:02.945 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:11:02.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:11:02.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:11:02.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:11:02.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:11:02.947 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:11:02.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:11:02.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:11:02.948 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-17T18:11:02.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:11:02.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:11:02.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:11:02.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:11:02.979 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:02.977+0000 7f79c381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:02.981 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:02.979+0000 7f79c381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:02.982 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:02.980+0000 7f79c381a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:11:03.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:03.374 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:11:03.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:11:03.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:03.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:11:03.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:04.078 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:04.076+0000 7f79c381a640 -1 Falling back to public interface 2024-09-17T18:11:04.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:04.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:04.809 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:11:04.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:11:04.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:04.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:11:05.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:06.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:06.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:06.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:11:06.253 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:11:06.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:06.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:11:06.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:07.485 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:07.483+0000 7f79c381a640 -1 osd.0 44 log_to_monitors true 2024-09-17T18:11:07.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:07.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:07.646 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:11:07.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:11:07.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:07.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:11:08.094 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up out weight 0 up_from 65 up_thru 35 down_at 55 last_clean_interval [5,44) [v2:127.0.0.1:6802/2027909831,v1:127.0.0.1:6803/2027909831] [v2:127.0.0.1:6804/2027909831,v1:127.0.0.1:6805/2027909831] exists,up bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-17T18:11:08.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:11:08.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:11:08.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:11:08.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:11:08.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:11:08.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:11:08.100 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:11:08.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:11:08.131 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:08.129+0000 7fb6eb80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:08.150 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:08.148+0000 7fb6eb80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:08.151 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:08.149+0000 7fb6eb80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:11:08.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:11:08.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:11:08.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:11:08.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:11:08.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:11:08.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:08.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:11:08.498 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:11:08.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:08.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:11:08.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:09.513 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:09.511+0000 7fb6eb80d640 -1 Falling back to public interface 2024-09-17T18:11:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:09.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:11:09.938 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:11:09.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:09.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:11:10.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:11.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:11.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:11.385 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:11:11.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:11:11.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:11.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:11:11.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:11:12.561 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:11:12.559+0000 7fb6eb80d640 -1 osd.2 44 log_to_monitors true 2024-09-17T18:11:12.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:11:12.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:11:12.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:11:12.792 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:11:12.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:11:12.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:11:13.270 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up out weight 0 up_from 69 up_thru 53 down_at 58 last_clean_interval [15,44) [v2:127.0.0.1:6810/4086742626,v1:127.0.0.1:6811/4086742626] [v2:127.0.0.1:6812/4086742626,v1:127.0.0.1:6813/4086742626] exists,up dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:11:13.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:11:13.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:11:13.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:11:13.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:13.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-17T18:11:13.959 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already in. 2024-09-17T18:11:13.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:11:13.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-17T18:11:15.041 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already in. 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:11:15.053 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:11:15.054 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:11:15.054 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:11:15.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:11:15.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:11:15.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:11:15.108 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:15.746 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:11:15.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874244 2024-09-17T18:11:15.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874244 2024-09-17T18:11:15.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-279172874244' 2024-09-17T18:11:15.959 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:15.959 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:11:16.098 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:11:16.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:11:16.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:11:16.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:11:16.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:16.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:11:16.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743427 2024-09-17T18:11:16.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743427 2024-09-17T18:11:16.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-279172874244 2-296352743427' 2024-09-17T18:11:16.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:11:16.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-279172874244 2024-09-17T18:11:16.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:11:16.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:11:16.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-279172874244 2024-09-17T18:11:16.271 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:11:16.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874244 2024-09-17T18:11:16.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 279172874244' 2024-09-17T18:11:16.273 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 279172874244 2024-09-17T18:11:16.273 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:11:16.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874242 -lt 279172874244 2024-09-17T18:11:16.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:11:17.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:11:17.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:11:18.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874244 -lt 279172874244 2024-09-17T18:11:18.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:11:18.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-296352743427 2024-09-17T18:11:18.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:11:18.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:11:18.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-296352743427 2024-09-17T18:11:18.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:11:18.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743427 2024-09-17T18:11:18.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-09-17T18:11:18.230 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 296352743427 2024-09-17T18:11:18.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:11:18.647 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743427 -lt 296352743427 2024-09-17T18:11:18.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:11:18.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:18.648 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:19.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-17T18:11:19.186 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:19.186 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:19.187 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:19.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:19.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:19.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:19.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:20.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:11:20.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:20.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:20.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:20.209 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:20.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:20.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:20.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:20.210 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:20.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:20.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:20.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:20.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:21.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:21.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:21.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:21.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:11:21.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:21.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:21.891 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:22.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:22.279 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:22.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:22.280 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:22.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:22.817 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:23.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:23.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:23.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:11:23.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:23.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:11:23.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:23.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:23.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:23.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:23.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:23.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:23.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:23.776 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:24.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:24.157 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:24.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:24.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:25.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:25.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:25.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:11:25.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:25.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:26.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:26.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:26.404 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:26.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:26.405 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:26.977 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:26.978 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:27.504 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:27.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:27.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:11:27.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:27.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:11:29.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:29.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:29.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:29.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:29.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:29.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:29.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:29.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:29.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:29.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:29.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:29.505 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:30.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:30.078 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:30.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:30.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:30.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:30.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:11:30.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:30.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:11:33.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:33.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:33.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:34.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:34.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:34.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:34.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:34.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:34.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:35.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:35.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:35.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:11:35.358 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:35.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:11:41.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:41.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:41.760 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:41.760 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:41.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:41.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:41.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:41.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:42.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:42.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:42.160 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:42.160 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:42.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:42.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:42.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:42.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:42.689 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:43.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:43.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:43.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:11:43.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:43.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:11:50.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:50.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:50.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:50.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:50.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:50.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:50.518 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:50.519 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:50.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:50.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:50.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:50.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:51.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:51.451 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:52.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:52.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:52.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:11:52.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:11:52.565 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:11:52.565 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "fa029d6e-8fa5-4845-a127-4f64dfa9eeee", 2024-09-17T18:11:52.565 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:11:52.562999+0000", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:11:52.566 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:11:52.567 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 98/294 objects degraded (33.333%), 32 pgs degraded", 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 32 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.568 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.569 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.570 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.571 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.572 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.573 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.574 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.575 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:11:52.576 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.577 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:08:22.968949Z", 2024-09-17T18:11:52.578 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:08:22.968949Z", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:11:52.579 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:11:52.580 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:11:52.581 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 77, 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:08:23.858917+0000", 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:11:15.733631+0000", 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:11:12.785029+0000", 2024-09-17T18:11:52.582 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:11:14.468014+0000", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 9, 2024-09-17T18:11:52.583 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:11:52.584 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:10:10.136580+0000", 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 1, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool", 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:11:52.585 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "on", 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 32, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 22, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 32, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 32, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 32, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.586 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "74", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "31", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 0, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 0, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:11:52.587 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:11:52.588 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:11:52.589 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {}, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "read_balance": { 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "score_type": "Fair distribution", 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "score_acting": 1.690000057220459, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "score_stable": 1.690000057220459, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_score": 0.67000001668930054, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_acting": 1.1299999952316284, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_stable": 1.1299999952316284, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity_weighted": 1, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity": 1, 2024-09-17T18:11:52.590 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity_weighted": 1 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "bfecdf44-7455-4b04-8a35-f511ef8dd599", 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:11:52.591 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 44, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 65, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 74, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 55, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.592 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.593 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.594 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:11:52.595 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/2027909831", 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/2027909831", 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2027909831", 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2027909831", 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:11:52.596 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "abefbbb4-172f-4bc4-b393-5e13b15cc9d5", 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 50, 2024-09-17T18:11:52.597 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 56, 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.598 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.599 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.600 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.601 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/1332764119", 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/1332764119", 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1332764119", 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1332764119", 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "dd3049bf-4a8b-4a5d-a1cc-256f71296e70", 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:11:52.602 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 44, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 69, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 75, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 58, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.603 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.604 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.605 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:11:52.606 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/4086742626", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/4086742626", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4086742626", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4086742626", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.607 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:00.150100+0000", 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:08:57.503362+0000", 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:11:52.608 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:00.666916+0000", 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:09:31.819443+0000", 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:01.292206+0000", 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:11:52.609 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:10:06.504219+0000", 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:11:52.610 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:11:52.611 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:11:52.612 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/2027909831,v1:127.0.0.1:6805/2027909831]", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:11:52.613 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:08:29.881823Z", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:11:52.614 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/2027909831,v1:127.0.0.1:6803/2027909831]", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2027909831,v1:127.0.0.1:6809/2027909831]", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2027909831,v1:127.0.0.1:6807/2027909831]", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:11:52.615 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/1332764119,v1:127.0.0.1:6813/1332764119]", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:11:52.616 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:11:52.617 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:09:03.902389Z", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:11:52.618 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/1332764119,v1:127.0.0.1:6811/1332764119]", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1332764119,v1:127.0.0.1:6817/1332764119]", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1332764119,v1:127.0.0.1:6815/1332764119]", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:11:52.619 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/4086742626,v1:127.0.0.1:6813/4086742626]", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:11:52.620 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.621 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:09:38.718764Z", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/4086742626,v1:127.0.0.1:6811/4086742626]", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4086742626,v1:127.0.0.1:6817/4086742626]", 2024-09-17T18:11:52.622 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4086742626,v1:127.0.0.1:6815/4086742626]", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.623 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [ 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: "floor": 34 2024-09-17T18:11:52.624 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 65 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 69 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.625 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 77, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.626 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:11:52.627 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.628 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:11:52.629 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.630 INFO:tasks.workunit.client.0.smithi157.stderr:report 1808423943 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:11:52.631 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.632 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:11:52.633 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.634 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.635 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.636 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.637 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.638 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:11:52.639 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:11:52.640 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:08:23:834288+0000", 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:11:52.641 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.642 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:11:52.643 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 32, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 32, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 46687, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 98, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 294, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:11:52.644 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 98, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 98, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 48, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 48, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:11:52.645 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:11:52.646 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:11:52.647 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 585, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 585, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 64, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 64, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:11:52.648 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 96, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 84440, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2864, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:11:52.649 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314488360, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036080640, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2932736, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 708186, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:11:52.650 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 110, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 110, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 110000000, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 110000000 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:11:52.651 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 96, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 84440, 2024-09-17T18:11:52.652 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2864, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314488360, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036080640, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2932736, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 708186, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:11:52.653 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 110, 2024-09-17T18:11:52.654 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 110, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 110000000, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 110000000 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 32, 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:11:52.655 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 46687, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 98, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 294, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 98, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 98, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 48, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 48, 2024-09-17T18:11:52.656 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:11:52.657 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:11:52.658 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1220608, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 143276, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 585, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 585, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 64, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 64, 2024-09-17T18:11:52.659 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 279172874251 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672990 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 296352743434 2024-09-17T18:11:52.660 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 32 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 18, 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 32, 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:11:52.661 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 14, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 32, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:11:52.662 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 5, 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:11:08.680178+0000", 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "services": { 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": { 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "daemons": { 2024-09-17T18:11:52.663 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": "", 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "0": { 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "start_epoch": 0, 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "start_stamp": "0.000000", 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "gid": 0, 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "metadata": {}, 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "task_status": {} 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:11:52.664 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 93, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 429, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 588, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:11:52.665 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:11:52.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:11:52.666 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-17T18:11:52.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-17T18:11:52.666 INFO:tasks.workunit.client.0.smithi157.stdout:writing non-divergent object existing_4 2024-09-17T18:11:52.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-17T18:11:52.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:11:52.967 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:11:52.967 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 77:132 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 77:132 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 77:124 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 77:94 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 77:131 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 77:123 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 77:152 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 77:138 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 77:130 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:11:52.968 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 77:144 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 77:122 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 77:142 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 77:137 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 77:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 36'61 77:122 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 77:124 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 77:132 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 77:119 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 77:134 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 77:113 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 77:146 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 77:121 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 77:120 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:11:52.970 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:11:52.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-17T18:11:53.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:11:53.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-17T18:11:53.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:11:53.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:11:53.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:11:53.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:11:53.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:11:53.098 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:11:53.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:11:53.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:11:53.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:11:53.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:11:53.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:11:53.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:11:53.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:11:53.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:11:53.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:53.566 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:11:53.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874254 2024-09-17T18:11:53.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874254 2024-09-17T18:11:53.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-279172874254' 2024-09-17T18:11:53.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:53.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:11:53.798 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:11:53.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:11:53.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:11:53.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:11:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:11:53.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:11:53.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743437 2024-09-17T18:11:53.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743437 2024-09-17T18:11:53.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-279172874254 2-296352743437' 2024-09-17T18:11:53.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:11:53.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-279172874254 2024-09-17T18:11:53.924 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:11:53.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:11:53.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-279172874254 2024-09-17T18:11:53.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:11:53.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874254 2024-09-17T18:11:53.926 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 279172874254 2024-09-17T18:11:53.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 279172874254' 2024-09-17T18:11:53.927 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:11:54.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874251 -lt 279172874254 2024-09-17T18:11:54.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:11:55.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:11:55.344 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:11:55.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874254 -lt 279172874254 2024-09-17T18:11:55.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:11:55.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-296352743437 2024-09-17T18:11:55.779 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:11:55.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:11:55.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-296352743437 2024-09-17T18:11:55.780 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:11:55.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743437 2024-09-17T18:11:55.781 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 296352743437' 2024-09-17T18:11:55.784 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 296352743437 2024-09-17T18:11:55.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:11:56.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743437 -lt 296352743437 2024-09-17T18:11:56.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:11:56.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:56.204 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:56.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:57.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:57.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:57.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:57.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:57.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:57.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:11:57.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:11:57.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:11:57.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:57.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:57.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:58.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:58.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:58.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:58.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:11:58.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:11:58.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:11:58.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:11:58.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:11:59.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:11:59.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:11:59.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:11:59.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:11:59.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:11:59.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:11:59.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:11:59.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:11:59.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:11:59.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:00.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:00.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:00.361 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:00.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:00.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:00.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:12:00.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:00.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:12:01.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:01.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:01.298 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:01.298 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:01.298 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:01.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:01.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:01.299 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:01.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:01.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:01.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:01.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:02.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:02.221 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:02.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:02.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:02.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:12:02.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:02.788 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:12:03.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:03.589 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:03.590 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:03.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:03.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:03.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:03.987 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:04.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:04.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:04.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:04.533 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:05.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:05.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:05.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:12:05.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:05.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:06.725 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:06.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:07.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:07.140 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:07.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:07.141 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:07.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:07.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:07.665 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:08.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:08.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:08.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:12:08.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:08.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:12:11.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:11.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:11.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:11.418 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:11.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:11.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:11.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:11.419 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:11.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:11.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:11.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:11.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:12.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:12.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:12.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:12.355 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:12.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:12.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:12.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:12:12.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:12.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:12:19.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:19.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:19.312 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:19.707 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:19.707 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:19.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:19.708 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:20.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:20.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:20.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:20.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:20.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:20.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:20.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:20.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:20.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:20.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:20.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:20.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:12:20.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:12:20.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:12:28.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:12:28.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:12:28.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:12:28.094 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:12:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:12:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:12:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:12:28.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:12:28.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:12:28.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:12:28.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:12:28.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:12:29.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-17T18:12:29.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:12:29.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:12:29.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:12:29.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:12:29.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:12:29.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:12:29.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:12:29.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:12:29.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:12:29.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:12:29.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:12:29.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "fa029d6e-8fa5-4845-a127-4f64dfa9eeee", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:12:30.130675+0000", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:12:30.133 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:12:30.134 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:12:30.135 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 98/294 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 64 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.136 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.5 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.137 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.9 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.138 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.139 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.140 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.15 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.17 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.18 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.19 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.141 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.1f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.143 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:12:30.144 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:08:22.968949Z", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:08:22.968949Z", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:12:30.145 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.146 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:12:30.147 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 77, 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:08:23.858917+0000", 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:11:15.733631+0000", 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:11:12.785029+0000", 2024-09-17T18:12:30.148 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:11:14.468014+0000", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 9, 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:12:30.149 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:10:10.136580+0000", 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 1, 2024-09-17T18:12:30.150 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool", 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 0, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "on", 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 32, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 22, 2024-09-17T18:12:30.151 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 32, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 32, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 32, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "74", 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:12:30.152 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "31", 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "selfmanaged", 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 0, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 0, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [], 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:12:30.153 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "", 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:12:30.154 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 0, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {}, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "read_balance": { 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "score_type": "Fair distribution", 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "score_acting": 1.690000057220459, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "score_stable": 1.690000057220459, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_score": 0.67000001668930054, 2024-09-17T18:12:30.155 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_acting": 1.1299999952316284, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "raw_score_stable": 1.1299999952316284, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity_weighted": 1, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity": 1, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "average_primary_affinity_weighted": 1 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "bfecdf44-7455-4b04-8a35-f511ef8dd599", 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:12:30.156 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 44, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 65, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 74, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 55, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.157 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.158 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 2027909831 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/2027909831", 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/2027909831", 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2027909831", 2024-09-17T18:12:30.161 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2027909831", 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "abefbbb4-172f-4bc4-b393-5e13b15cc9d5", 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:12:30.162 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 50, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 56, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.163 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.164 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.165 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.166 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1332764119 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/1332764119", 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/1332764119", 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1332764119", 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1332764119", 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.167 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "dd3049bf-4a8b-4a5d-a1cc-256f71296e70", 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 44, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 69, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 75, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 58, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.168 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.169 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.170 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:12:30.171 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 4086742626 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/4086742626", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/4086742626", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4086742626", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4086742626", 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:12:30.172 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:00.150100+0000", 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:12:30.173 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:08:57.503362+0000", 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:00.666916+0000", 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:09:31.819443+0000", 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:12:30.174 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:11:01.292206+0000", 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:12:30.175 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:10:06.504219+0000", 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:12:30.176 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:12:30.177 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/2027909831,v1:127.0.0.1:6805/2027909831]", 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:12:30.178 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:12:30.179 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:08:29.881823Z", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/2027909831,v1:127.0.0.1:6803/2027909831]", 2024-09-17T18:12:30.180 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2027909831,v1:127.0.0.1:6809/2027909831]", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2027909831,v1:127.0.0.1:6807/2027909831]", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:12:30.181 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/1332764119,v1:127.0.0.1:6813/1332764119]", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:12:30.182 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:12:30.183 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:09:03.902389Z", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/1332764119,v1:127.0.0.1:6811/1332764119]", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1332764119,v1:127.0.0.1:6817/1332764119]", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1332764119,v1:127.0.0.1:6815/1332764119]", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:12:30.184 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:12:30.185 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/4086742626,v1:127.0.0.1:6813/4086742626]", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:12:30.186 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:09:38.718764Z", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:12:30.187 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/4086742626,v1:127.0.0.1:6811/4086742626]", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4086742626,v1:127.0.0.1:6817/4086742626]", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4086742626,v1:127.0.0.1:6815/4086742626]", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:12:30.188 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [ 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: "floor": 34 2024-09-17T18:12:30.189 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 65 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:12:30.190 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 69 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stderr:report 1276857751 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:12:30.191 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 77, 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.192 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.193 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:12:30.194 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:12:30.195 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:12:30.196 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.197 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.198 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.199 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:12:30.200 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:12:30.201 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:12:30.202 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:12:30.203 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:12:30.204 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 0, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.205 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:08:23:834288+0000", 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:12:30.206 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:12:30.207 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 32, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 32, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:12:30.208 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 46358, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 98, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 294, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 98, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 98, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:12:30.209 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 49, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 49, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:12:30.210 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.211 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 586, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 586, 2024-09-17T18:12:30.212 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 64, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 64, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 96, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 84408, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2864, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:12:30.213 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314488392, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036113408, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2932736, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 707528, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:12:30.214 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 110, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 110, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 110000000, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 110000000 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:12:30.215 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 96, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 84408, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2864, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:12:30.216 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314488392, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036113408, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2932736, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 707528, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10396, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416932 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:12:30.217 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 110, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 110, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 110000000, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 110000000 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:12:30.218 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 32, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 46358, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 98, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 0, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 294, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 98, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:12:30.219 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 98, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 49, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 49, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:12:30.220 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:12:30.221 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1220608, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 142618, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:12:30.222 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 586, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 586, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 64, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 64, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 279172874261 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.223 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672990 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 296352743444 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 32 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.224 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 18, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 32, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 14, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 32, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.225 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 7, 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:12:14.696573+0000", 2024-09-17T18:12:30.226 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 112, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 495, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:12:30.227 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 673, 2024-09-17T18:12:30.228 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:12:30.228 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:12:30.228 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:12:30.228 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:12:30.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:12:30.229 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery 2024-09-17T18:12:30.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-17T18:12:30.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 77:132 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 77:132 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 77:124 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 77:94 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 77:131 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 77:123 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.530 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 77:152 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 77:138 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 77:130 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 77:144 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 77:122 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 77:142 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 77:137 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 77:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 77:123 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:12:18.606680+0000 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 77:124 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.531 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 77:132 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 77:119 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 77:134 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 77:113 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 77:146 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 77:121 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 77:120 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:30.532 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:30.533 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:30.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:12:30.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:12:30.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok set_recovery_delay 100000 2024-09-17T18:12:30.623 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:12:30.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:12:30.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:12:30.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok set_recovery_delay 100000 2024-09-17T18:12:30.712 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:12:30.719 INFO:tasks.workunit.client.0.smithi157.stdout:revive divergent 1 2024-09-17T18:12:30.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-17T18:12:30.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:31.089 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:31.089 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 77:132 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:31.089 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 77:132 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 77:124 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 77:94 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 77:131 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 77:123 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 77:152 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 77:138 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 77:130 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 77:144 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 77:122 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.090 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 77:142 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 77:137 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 77:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 77:123 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:12:18.606680+0000 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 77:124 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 77:132 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 77:101 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 77:119 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 77:134 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 77:113 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 77:146 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:31.091 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 77:111 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 77:134 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 77:121 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 77:120 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 77:129 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:31.092 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:31.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-17T18:12:31.750 INFO:tasks.workunit.client.0.smithi157.stderr:noup is set 2024-09-17T18:12:31.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-17T18:12:31.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:12:31.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:12:31.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:12:31.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:12:31.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:12:31.772 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:12:31.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:12:31.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:12:31.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:12:31.773 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:12:31.774 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:12:31.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:12:31.799 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:31.798+0000 7f8fc920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:31.809 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:31.808+0000 7f8fc920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:31.810 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:31.809+0000 7f8fc920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:32.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-17T18:12:32.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-17T18:12:32.908 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:32.907+0000 7f8fc920d640 -1 Falling back to public interface 2024-09-17T18:12:35.845 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:35.843+0000 7f8fc920d640 -1 osd.1 54 log_to_monitors true 2024-09-17T18:12:37.233 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery divergent 2024-09-17T18:12:37.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-17T18:12:37.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 79:135 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 79:137 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 79:135 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 78:126 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.747896+0000 0 0 2024-09-17T18:12:37.620 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 79:97 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:22.636647+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 79:134 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 79:126 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.607403+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 79:114 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.667215+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 78:154 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 79:141 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 78:103 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.740341+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 79:133 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 78:146 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 78:124 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:24.749559+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 79:145 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 78:103 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.764058+0000 0 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 78:139 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 79:131 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 79:126 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.615525+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 79:127 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.642997+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 78:134 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:19.726755+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 78:103 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.765470+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 78:121 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 78:131 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.713536+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 79:137 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 78:115 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:25.749831+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 78:148 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 79:114 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.626634+0000 0 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 79:137 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 79:124 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 79:123 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:37.622 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 78:131 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:23.709395+0000 0 0 2024-09-17T18:12:37.623 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:37.623 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:37.623 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:12:37.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok set_recovery_delay 100000 2024-09-17T18:12:37.713 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:12:37.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-17T18:12:38.347 INFO:tasks.workunit.client.0.smithi157.stderr:noup is unset 2024-09-17T18:12:38.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-17T18:12:38.359 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:12:38.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:12:38.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:12:38.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:12:38.928 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 65 up_thru 82 down_at 55 last_clean_interval [5,44) [v2:127.0.0.1:6802/2027909831,v1:127.0.0.1:6803/2027909831] [v2:127.0.0.1:6804/2027909831,v1:127.0.0.1:6805/2027909831] exists,up bfecdf44-7455-4b04-8a35-f511ef8dd599 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:12:38.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:12:38.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:12:38.931 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:12:38.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:12:38.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:12:39.512 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 56 last_clean_interval [10,54) [v2:127.0.0.1:6818/2144736861,v1:127.0.0.1:6819/2144736861] [v2:127.0.0.1:6820/2144736861,v1:127.0.0.1:6821/2144736861] exists,up abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:12:39.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:12:39.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:12:39.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:12:39.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:12:39.516 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:12:39.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:12:39.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:12:39.934 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 69 up_thru 82 down_at 58 last_clean_interval [15,44) [v2:127.0.0.1:6810/4086742626,v1:127.0.0.1:6811/4086742626] [v2:127.0.0.1:6812/4086742626,v1:127.0.0.1:6813/4086742626] exists,up dd3049bf-4a8b-4a5d-a1cc-256f71296e70 2024-09-17T18:12:39.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:12:39.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:12:39.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:12:39.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:40.416 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:40.416 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 80:137 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:40.416 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:40.416 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 80:137 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 80:130 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.747896+0000 0 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 80:99 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:22.636647+0000 0 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 80:136 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 80:128 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.607403+0000 0 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.667215+0000 0 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 80:158 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 80:143 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.740341+0000 0 0 2024-09-17T18:12:40.417 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 80:135 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 80:150 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 80:128 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:24.749559+0000 0 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 80:147 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.764058+0000 0 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 80:143 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 80:133 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:40.418 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 80:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.615525+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 80:129 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.642997+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 80:138 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:19.726755+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.765470+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 80:125 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.713536+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 80:139 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 80:119 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:25.749831+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 80:152 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.626634+0000 0 0 2024-09-17T18:12:40.419 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 80:126 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 80:125 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:23.709395+0000 0 0 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:40.420 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:40.427 INFO:tasks.workunit.client.0.smithi157.stdout:wait for peering 2024-09-17T18:12:40.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-17T18:12:40.428 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 80:137 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 80:137 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 80:130 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.747896+0000 0 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 80:99 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:22.636647+0000 0 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 80:136 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 80:128 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.607403+0000 0 0 2024-09-17T18:12:40.804 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.667215+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 80:158 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 80:143 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.740341+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 80:135 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 80:150 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 80:128 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:24.749559+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 80:147 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.764058+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 80:143 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 80:133 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 80:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.615525+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 80:129 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.642997+0000 0 0 2024-09-17T18:12:40.805 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 80:138 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:19.726755+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.765470+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 80:125 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.713536+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 80:139 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 80:119 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:25.749831+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 80:152 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.626634+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 80:126 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 80:125 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:23.709395+0000 0 0 2024-09-17T18:12:40.806 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:40.807 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:40.807 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:40.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-17T18:12:40.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-17T18:12:40.962 INFO:tasks.workunit.client.0.smithi157.stdout:killing divergent 1 2024-09-17T18:12:40.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:41.402 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:41.402 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373415+0000 32'60 80:137 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:15.407670+0000 3 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373819+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:07:54.675878+0000 2 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374306+0000 32'61 80:137 [2,0] 2 [2,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:33:25.355782+0000 4 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655498+0000 38'60 80:130 [0,2] 0 [0,2] 0 38'60 2024-09-17T18:10:46.112135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.747896+0000 0 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374436+0000 30'59 80:99 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:41.835556+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:22.636647+0000 0 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374989+0000 36'61 80:136 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:43.909443+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:28:34.716269+0000 5 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 4 0 0 1857 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374151+0000 38'61 80:128 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:20.607403+0000 0 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.373500+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.667215+0000 0 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 4 0 4 0 0 1932 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:15.654709+0000 34'62 80:158 [0,2] 0 [0,2] 0 34'62 2024-09-17T18:10:43.782481+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:56:23.260097+0000 4 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 6 0 0 2898 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.374152+0000 36'62 80:143 [2,0] 2 [2,0] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:26:37.333639+0000 6 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654615+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.740341+0000 0 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 5 0 0 2415 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373449+0000 39'61 80:135 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:04:54.036254+0000 4 0 2024-09-17T18:12:41.403 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 1 0 0 483 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654628+0000 32'60 80:150 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:56.214479+0000 1 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.656003+0000 35'60 80:128 [0,2] 0 [0,2] 0 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:24.749559+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 2 0 0 966 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373811+0000 34'61 80:147 [2,0] 2 [2,0] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:12:20.319289+0000 2 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 1 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.654708+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:16.764058+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 3 0 0 1449 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.654613+0000 38'60 80:143 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:44:06.439062+0000 2 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.373797+0000 36'61 80:133 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:55:48.087987+0000 3 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 0 4 0 0 1528 0 0 20 42 20 active+undersized+degraded 2024-09-17T18:11:16.373479+0000 77'62 80:128 [2,0] 2 [2,0] 2 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.615525+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.373493+0000 32'60 80:129 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:17.642997+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 3 0 0 1374 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.655118+0000 39'60 80:138 [0,2] 0 [0,2] 0 39'60 2024-09-17T18:10:45.154491+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:19.726755+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 2 0 0 891 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655015+0000 30'59 80:107 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.765470+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 1 0 0 483 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:15.655005+0000 30'59 80:125 [0,2] 0 [0,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T02:04:40.184116+0000 1 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655183+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:18.713536+0000 0 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.287083+0000 39'61 80:139 [2,0] 2 [2,0] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:18:38.344956+0000 2 0 2024-09-17T18:12:41.404 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 3 0 0 1373 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:15.951811+0000 32'60 80:119 [0,2] 0 [0,2] 0 32'60 2024-09-17T18:10:42.935474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:25.749831+0000 0 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 4 0 0 1932 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.591621+0000 32'61 80:152 [0,2] 0 [0,2] 0 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:09:12.791082+0000 4 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 2 0 0 890 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374468+0000 30'59 80:116 [2,0] 2 [2,0] 2 0'0 2024-09-17T18:10:10.186768+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:21.626634+0000 0 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 3 0 0 1449 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:16.374642+0000 39'61 80:139 [2,0] 2 [2,0] 2 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:43:56.267666+0000 2 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 2 0 0 966 0 0 17 42 17 active+undersized+degraded 2024-09-17T18:11:16.374793+0000 30'59 80:126 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:39.938495+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:29:58.267654+0000 2 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 4 0 0 1932 0 0 18 42 18 active+undersized+degraded 2024-09-17T18:11:16.374185+0000 34'60 80:125 [2,0] 2 [2,0] 2 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:42:55.401193+0000 3 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 0 4 0 0 1856 0 0 19 42 19 active+undersized+degraded 2024-09-17T18:11:15.655597+0000 39'61 80:135 [0,2] 0 [0,2] 0 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 user requested scrub scheduled @ 2024-09-17T18:13:23.709395+0000 0 0 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:41.405 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:41.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-17T18:12:41.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:12:41.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:12:41.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:12:41.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:12:41.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.33174.out 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.33174.out 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:12:41.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:12:41.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-17T18:12:41.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.33174.out 2024-09-17T18:12:43.292 INFO:tasks.workunit.client.0.smithi157.stderr:Exporting 1.1b info 1.1b( v 39'61 lc 0'0 (29'42,39'61] local-lis/les=82/83 n=4 ec=31/22 lis/c=82/50 les/c/f=83/51/0 sis=82) 2024-09-17T18:12:43.292 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-17T18:12:43.304 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-17T18:12:43.305 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-17T18:12:43.311 INFO:tasks.workunit.client.0.smithi157.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-17T18:12:43.312 INFO:tasks.workunit.client.0.smithi157.stderr:Export successful 2024-09-17T18:12:43.312 INFO:tasks.workunit.client.0.smithi157.stdout: marking collection for removal 2024-09-17T18:12:43.312 INFO:tasks.workunit.client.0.smithi157.stdout:setting '_remove' omap key 2024-09-17T18:12:43.312 INFO:tasks.workunit.client.0.smithi157.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-17T18:12:43.408 INFO:tasks.workunit.client.0.smithi157.stderr:Remove successful 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33174.out 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:12:43.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-17T18:12:43.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33174.out 2024-09-17T18:12:45.139 INFO:tasks.workunit.client.0.smithi157.stdout:get_pg_num_history pg_num_history pg_num_history(e84 pg_nums {1={22=1,31=32}} deleted_pools ) 2024-09-17T18:12:45.139 INFO:tasks.workunit.client.0.smithi157.stdout:Importing pgid 1.1b 2024-09-17T18:12:45.139 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-17T18:12:45.140 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:12:45.203 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-17T18:12:45.204 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:12:45.237 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-17T18:12:45.237 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:12:45.270 INFO:tasks.workunit.client.0.smithi157.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-17T18:12:45.270 INFO:tasks.workunit.client.0.smithi157.stdout:snapset 0={} 2024-09-17T18:12:45.303 INFO:tasks.workunit.client.0.smithi157.stdout:write_pg epoch 83 info 1.1b( v 39'61 lc 0'0 (29'42,39'61] local-lis/les=82/83 n=4 ec=31/22 lis/c=82/50 les/c/f=83/51/0 sis=82) 2024-09-17T18:12:45.345 INFO:tasks.workunit.client.0.smithi157.stdout:Import successful 2024-09-17T18:12:45.861 INFO:tasks.workunit.client.0.smithi157.stdout:reviving divergent 1 2024-09-17T18:12:45.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-17T18:12:45.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:39.672406+0000 32'60 84:150 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:13:55.651728+0000 3 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:39.210944+0000 39'61 84:151 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:52:48.440232+0000 2 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:39.671488+0000 32'61 84:150 [2,1,0] 2 [2,1,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:46:35.870517+0000 4 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:40.926502+0000 38'60 84:181 [1,0,2] 1 [1,0,2] 1 38'60 2024-09-17T18:12:40.926456+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:19:06.226596+0000 3 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:12:39.211343+0000 30'59 84:141 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:55.247230+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:43:09.537666+0000 3 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 4 1 1 0 0 1857 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:39.212788+0000 36'61 84:166 [1,2,0] 1 [1,2,0] 1 45'62 2024-09-17T18:10:52.219421+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 4 0 0 0 0 1857 0 0 19 42 19 active+clean 2024-09-17T18:12:39.211004+0000 38'61 84:172 [1,2,0] 1 [1,2,0] 1 38'61 2024-09-17T18:10:53.240404+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 891 0 0 17 42 17 active+clean 2024-09-17T18:12:39.213082+0000 30'59 84:153 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:51.192459+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 5 0 0 0 0 2415 0 0 21 42 21 active+clean 2024-09-17T18:12:39.208017+0000 84'63 84:163 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:56.202865+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:18:23.362162+0000 4 0 2024-09-17T18:12:46.245 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 active+clean 2024-09-17T18:12:39.672490+0000 36'62 84:156 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:42:36.458709+0000 6 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 891 0 0 17 42 17 active+clean 2024-09-17T18:12:39.212204+0000 30'59 84:120 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:34.293468+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:12:39.168121+0000 39'61 84:137 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T05:44:08.125706+0000 4 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 active+clean 2024-09-17T18:12:39.207711+0000 32'60 84:165 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:28:16.063528+0000 1 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1374 0 0 18 42 18 active+clean 2024-09-17T18:12:39.212802+0000 35'60 84:136 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:10:41.139505+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 active+clean 2024-09-17T18:12:39.672633+0000 34'61 84:160 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:29:26.200249+0000 2 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 1 0 0 0 0 890 0 0 17 42 17 active+clean 2024-09-17T18:12:39.212221+0000 30'59 84:130 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:38.249849+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:39.210100+0000 38'60 84:137 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:48:57.084661+0000 2 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:39.671364+0000 36'61 84:146 [2,1,0] 2 [2,1,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:37:00.477703+0000 3 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 1 1 0 0 1528 0 0 20 42 20 active+recovering+degraded 2024-09-17T18:12:39.213365+0000 77'62 84:154 [1,2,0] 1 [1,2,0] 1 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:39.831177+0000 32'60 84:150 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:12:39.831124+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:40:44.111577+0000 4 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:39.673640+0000 39'60 84:161 [0,2,1] 0 [0,2,1] 0 39'60 2024-09-17T18:12:39.673595+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:09:46.849303+0000 3 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 1 0 0 891 0 0 17 42 17 active+recovery_wait+degraded 2024-09-17T18:12:39.207918+0000 30'59 84:120 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:12:46.246 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:12:39.207418+0000 30'59 84:140 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:33:58.646885+0000 1 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 4 0 0 0 0 1856 0 0 19 42 19 active+clean 2024-09-17T18:12:39.212471+0000 39'61 84:139 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:44.076606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:39.603811+0000 39'61 84:152 [2,0,1] 2 [2,0,1] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T23:33:26.214056+0000 2 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:39.923221+0000 32'60 84:122 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:12:39.923167+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:03:36.796427+0000 3 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:39.131228+0000 32'61 84:144 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:57:19.849329+0000 4 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 2 0 0 0 0 890 0 0 17 42 17 active+clean 2024-09-17T18:12:39.211775+0000 30'59 84:148 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:49.259560+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:39.211926+0000 39'61 84:163 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T03:01:28.292819+0000 2 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 890 0 0 17 42 17 active+clean 2024-09-17T18:12:39.211711+0000 30'59 84:136 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:54.238827+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 queued for scrub 0 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:39.208509+0000 34'60 84:138 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:09:21.721311+0000 3 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 1 1 0 0 1856 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:39.212305+0000 39'61 84:173 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:12:46.247 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:12:46.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:12:46.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:12:46.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:12:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:12:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:12:46.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:12:46.262 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:12:46.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:12:46.294 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:46.292+0000 7f1189a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:46.295 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:46.294+0000 7f1189a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:46.296 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:46.295+0000 7f1189a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:12:46.694 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:12:46.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:12:46.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:12:46.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:12:47.084 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 56 last_clean_interval [10,54) [v2:127.0.0.1:6818/2144736861,v1:127.0.0.1:6819/2144736861] [v2:127.0.0.1:6820/2144736861,v1:127.0.0.1:6821/2144736861] exists,up abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:12:47.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:12:47.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:12:47.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:12:47.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:12:47.086 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:12:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:12:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:12:47.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:12:47.512 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 82 up_thru 82 down_at 56 last_clean_interval [10,54) [v2:127.0.0.1:6818/2144736861,v1:127.0.0.1:6819/2144736861] [v2:127.0.0.1:6820/2144736861,v1:127.0.0.1:6821/2144736861] exists,up abefbbb4-172f-4bc4-b393-5e13b15cc9d5 2024-09-17T18:12:47.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:12:47.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:12:47.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:12:47.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-17T18:12:47.636 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:47.634+0000 7f1189a14640 -1 Falling back to public interface 2024-09-17T18:12:51.076 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:12:51.074+0000 7f1189a14640 -1 osd.1 84 log_to_monitors true 2024-09-17T18:13:07.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-17T18:13:07.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:13:07.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:13:07.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:13:07.515 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:07.516 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:07.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:13:07.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-17T18:13:07.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T18:13:07.602 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:13:07.602 INFO:tasks.workunit.client.0.smithi157.stdout: "ops": [], 2024-09-17T18:13:07.602 INFO:tasks.workunit.client.0.smithi157.stdout: "num_ops": 0 2024-09-17T18:13:07.603 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:13:07.610 INFO:tasks.workunit.client.0.smithi157.stdout:allowing recovery 2024-09-17T18:13:07.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-17T18:13:07.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.141663+0000 32'60 89:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:10:48.354209+0000 3 0 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.148104+0000 39'61 89:162 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T21:56:55.140762+0000 2 0 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:53.141955+0000 32'61 89:164 [2,1,0] 2 [2,1,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:51:37.200001+0000 4 0 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.146610+0000 38'60 89:192 [1,0,2] 1 [1,0,2] 1 38'60 2024-09-17T18:12:40.926456+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:21:00.498819+0000 3 0 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:12:53.147538+0000 30'59 89:152 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:55.247230+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:16:45.385158+0000 3 0 2024-09-17T18:13:07.989 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 4 1 1 0 0 1857 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:53.147247+0000 36'61 89:175 [1,2,0] 1 [1,2,0] 1 45'62 2024-09-17T18:10:52.219421+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:57.142602+0000 38'61 89:191 [1,2,0] 1 [1,2,0] 1 38'61 2024-09-17T18:12:57.142538+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:30:11.083851+0000 3 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:56.179106+0000 30'59 89:172 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:56.179045+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:18:38.551569+0000 2 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 5 0 0 0 0 2415 0 0 21 42 21 active+clean 2024-09-17T18:12:53.148548+0000 84'63 89:175 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:56.202865+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:14:45.454848+0000 4 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 active+clean 2024-09-17T18:12:53.140053+0000 36'62 89:170 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:59:09.062152+0000 6 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:13:01.117836+0000 30'59 89:139 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:13:01.117769+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:35:39.985270+0000 2 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:12:53.147383+0000 39'61 89:148 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:22:18.571230+0000 4 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 active+clean 2024-09-17T18:12:53.141659+0000 32'60 89:180 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:36:08.813823+0000 1 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:59.122070+0000 35'60 89:155 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:12:59.122010+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:20:05.979983+0000 3 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 active+clean 2024-09-17T18:12:53.141041+0000 34'61 89:174 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:40:07.490333+0000 2 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:54.142070+0000 30'59 89:149 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:12:54.142012+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:09:36.977815+0000 2 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.149065+0000 38'60 89:148 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:44:14.192445+0000 2 0 2024-09-17T18:13:07.990 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.140098+0000 36'61 89:160 [2,1,0] 2 [2,1,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:38:35.788527+0000 3 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 1 1 0 0 1528 0 0 20 42 20 active+recovery_wait+degraded 2024-09-17T18:12:53.149770+0000 77'62 89:162 [1,2,0] 1 [1,2,0] 1 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:53.142152+0000 32'60 89:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:12:39.831124+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:33:27.090516+0000 4 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.140301+0000 39'60 89:176 [0,2,1] 0 [0,2,1] 0 39'60 2024-09-17T18:12:39.673595+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:24:32.408915+0000 3 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 1 0 0 891 0 0 17 42 17 active+recovering+degraded 2024-09-17T18:12:53.149831+0000 30'59 89:134 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:12:53.140996+0000 30'59 89:155 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:17:58.095574+0000 1 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:13:00.116932+0000 39'61 89:158 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:13:00.116873+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:49:45.758179+0000 3 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.142636+0000 39'61 89:166 [2,0,1] 2 [2,0,1] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:46:58.136849+0000 2 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.149143+0000 32'60 89:132 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:12:39.923167+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:57:49.513358+0000 3 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:53.148665+0000 32'61 89:155 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T04:23:08.018318+0000 4 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:12:55.161836+0000 30'59 89:167 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:55.161770+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:27:58.626539+0000 1 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.149478+0000 39'61 89:174 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:19:14.641213+0000 2 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:58.133791+0000 30'59 89:155 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:58.133725+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:26:20.229023+0000 2 0 2024-09-17T18:13:07.991 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:53.147941+0000 34'60 89:149 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:26:54.069387+0000 3 0 2024-09-17T18:13:07.992 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 1 1 0 0 1856 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:53.148420+0000 39'61 89:182 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:07.992 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:13:07.992 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:13:07.992 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:13:08.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-17T18:13:08.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-17T18:13:08.112 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:13:08.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-17T18:13:08.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-17T18:13:08.235 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:13:08.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-17T18:13:08.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-17T18:13:08.358 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:13:08.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-17T18:13:08.368 INFO:tasks.workunit.client.0.smithi157.stdout:reading divergent objects 2024-09-17T18:13:08.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.1f 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.141663+0000 32'60 89:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:10:42.170983+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:10:48.354209+0000 3 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.1e 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.148104+0000 39'61 89:162 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.768221+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T21:56:55.140762+0000 2 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.1d 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:53.141955+0000 32'61 89:164 [2,1,0] 2 [2,1,0] 2 32'61 2024-09-17T18:10:38.906474+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:51:37.200001+0000 4 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.b 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.146610+0000 38'60 89:192 [1,0,2] 1 [1,0,2] 1 38'60 2024-09-17T18:12:40.926456+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:21:00.498819+0000 3 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.a 3 0 0 0 0 1449 0 0 17 42 17 active+clean 2024-09-17T18:12:53.147538+0000 30'59 89:152 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:55.247230+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T20:16:45.385158+0000 3 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.9 4 1 1 0 0 1857 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:53.147247+0000 36'61 89:175 [1,2,0] 1 [1,2,0] 1 45'62 2024-09-17T18:10:52.219421+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.8 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:57.142602+0000 38'61 89:191 [1,2,0] 1 [1,2,0] 1 38'61 2024-09-17T18:12:57.142538+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T19:30:11.083851+0000 3 0 2024-09-17T18:13:08.744 INFO:tasks.workunit.client.0.smithi157.stdout:1.7 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:56.179106+0000 30'59 89:172 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:56.179045+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:18:38.551569+0000 2 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.6 5 0 0 0 0 2415 0 0 21 42 21 active+clean 2024-09-17T18:12:53.148548+0000 84'63 89:175 [1,0,2] 1 [1,0,2] 1 34'62 2024-09-17T18:10:56.202865+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:14:45.454848+0000 4 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.5 6 0 0 0 0 2898 0 0 20 42 20 active+clean 2024-09-17T18:12:53.140053+0000 36'62 89:170 [2,0,1] 2 [2,0,1] 2 36'62 2024-09-17T18:10:41.013606+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:59:09.062152+0000 6 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.4 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:13:01.117836+0000 30'59 89:139 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:13:01.117769+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:35:39.985270+0000 2 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.3 5 0 0 0 0 2415 0 0 19 42 19 active+clean 2024-09-17T18:12:53.147383+0000 39'61 89:148 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:33.312120+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:22:18.571230+0000 4 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.2 1 0 0 0 0 483 0 0 18 42 18 active+clean 2024-09-17T18:12:53.141659+0000 32'60 89:180 [0,1,2] 0 [0,1,2] 0 32'60 2024-09-17T18:10:44.803972+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:36:08.813823+0000 1 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.c 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:59.122070+0000 35'60 89:155 [1,0,2] 1 [1,0,2] 1 35'60 2024-09-17T18:12:59.122010+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T01:20:05.979983+0000 3 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.1 2 0 0 0 0 966 0 0 19 42 19 active+clean 2024-09-17T18:12:53.141041+0000 34'61 89:174 [2,0,1] 2 [2,0,1] 2 34'61 2024-09-17T18:10:44.938731+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T02:40:07.490333+0000 2 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.1c 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:54.142070+0000 30'59 89:149 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:12:54.142012+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T05:09:36.977815+0000 2 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.149065+0000 38'60 89:148 [1,0,2] 1 [1,0,2] 1 30'59 2024-09-17T18:10:32.273207+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T19:44:14.192445+0000 2 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.d 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.140098+0000 36'61 89:160 [2,1,0] 2 [2,1,0] 2 36'61 2024-09-17T18:10:41.995080+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:38:35.788527+0000 3 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.e 4 1 1 0 0 1528 0 0 20 42 20 active+recovery_wait+degraded 2024-09-17T18:12:53.149770+0000 77'62 89:162 [1,2,0] 1 [1,2,0] 1 36'61 2024-09-17T18:10:48.198823+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.f 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:53.142152+0000 32'60 89:164 [2,0,1] 2 [2,0,1] 2 32'60 2024-09-17T18:12:39.831124+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T22:33:27.090516+0000 4 0 2024-09-17T18:13:08.745 INFO:tasks.workunit.client.0.smithi157.stdout:1.10 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.140301+0000 39'60 89:176 [0,2,1] 0 [0,2,1] 0 39'60 2024-09-17T18:12:39.673595+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:24:32.408915+0000 3 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.11 2 0 1 0 0 891 0 0 17 42 17 active+recovering+degraded 2024-09-17T18:12:53.149831+0000 30'59 89:134 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:36.883135+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.12 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:12:53.140996+0000 30'59 89:155 [0,1,2] 0 [0,1,2] 0 30'59 2024-09-17T18:10:38.797815+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:17:58.095574+0000 1 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.13 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:13:00.116932+0000 39'61 89:158 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:13:00.116873+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:49:45.758179+0000 3 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.14 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.142636+0000 39'61 89:166 [2,0,1] 2 [2,0,1] 2 33'60 2024-09-17T18:10:40.157392+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:46:58.136849+0000 2 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.15 3 0 0 0 0 1449 0 0 18 42 18 active+clean 2024-09-17T18:12:53.149143+0000 32'60 89:132 [1,0,2] 1 [1,0,2] 1 32'60 2024-09-17T18:12:39.923167+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:57:49.513358+0000 3 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.16 4 0 0 0 0 1932 0 0 19 42 19 active+clean 2024-09-17T18:12:53.148665+0000 32'61 89:155 [1,0,2] 1 [1,0,2] 1 32'61 2024-09-17T18:10:47.151060+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 periodic scrub scheduled @ 2024-09-19T04:23:08.018318+0000 4 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.17 1 0 0 0 0 483 0 0 17 42 17 active+clean 2024-09-17T18:12:55.161836+0000 30'59 89:167 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:55.161770+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T21:27:58.626539+0000 1 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.18 3 0 0 0 0 1449 0 0 19 42 19 active+clean 2024-09-17T18:12:53.149478+0000 39'61 89:174 [1,2,0] 1 [1,2,0] 1 32'60 2024-09-17T18:10:39.284284+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T01:19:14.641213+0000 2 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.19 2 0 0 0 0 966 0 0 17 42 17 active+clean 2024-09-17T18:12:58.133791+0000 30'59 89:155 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:12:58.133725+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-19T00:26:20.229023+0000 2 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.1a 4 0 0 0 0 1932 0 0 18 42 18 active+clean 2024-09-17T18:12:53.147941+0000 34'60 89:149 [1,2,0] 1 [1,2,0] 1 30'59 2024-09-17T18:10:35.243676+0000 0'0 2024-09-17T18:10:10.186768+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:26:54.069387+0000 3 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:1.1b 4 1 1 0 0 1856 0 0 19 42 19 active+recovery_wait+degraded 2024-09-17T18:12:53.148420+0000 39'61 89:182 [1,0,2] 1 [1,0,2] 1 39'61 2024-09-17T18:10:43.099707+0000 0'0 2024-09-17T18:10:10.186768+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:13:08.746 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:13:08.747 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:13:08.755 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-17T18:13:08.756 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-17T18:13:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:13:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-17T18:13:08.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:13:08.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-17T18:13:08.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:13:08.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-17T18:13:08.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:13:08.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-17T18:13:08.957 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-17T18:13:08.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:13:08.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-17T18:13:08.990 INFO:tasks.workunit.client.0.smithi157.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-17T18:13:08.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-17T18:13:08.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-17T18:13:09.040 INFO:tasks.workunit.client.0.smithi157.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-17T18:13:09.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-17T18:13:09.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-17T18:13:09.044 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:13:09.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 20 _merge_object_divergent_entries: keeping 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.065 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 45'60 last_divergent_update: 45'60 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 45'60 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.0.log:2024-09-17T18:12:38.611+0000 7f79a61c9640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'42 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 45'63 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-17T18:10:49.345183+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 20 _merge_object_divergent_entries: keeping 45'63 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-17T18:10:49.345183+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 45'63 last_divergent_update: 45'63 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-17T18:13:09.066 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 45'63 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 45'63 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-17T18:10:49.345183+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'42 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 44'62 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-09-17T18:10:49.283303+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 20 _merge_object_divergent_entries: keeping 44'62 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-09-17T18:10:49.283303+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 44'62 last_divergent_update: 44'62 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.611+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 45'62 (29'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-17T18:10:49.342509+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: keeping 45'62 (29'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-17T18:10:49.342509+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.067 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 45'62 last_divergent_update: 45'62 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 45'62 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 45'62 (29'2) modify 1:97db2a0f:::existing_2:head by client.4636.0:1 2024-09-17T18:10:49.342509+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'42 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 45'62 (29'4) delete 1:70b3bf12:::existing_4:head by client.4639.0:1 2024-09-17T18:10:49.343610+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: keeping 45'62 (29'4) delete 1:70b3bf12:::existing_4:head by client.4639.0:1 2024-09-17T18:10:49.343610+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 45'62 last_divergent_update: 45'62 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: more recent entry found: 77'62 (29'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-09-17T18:11:53.011225+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false, already merged 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 45'63 (0'0) modify 1:75ac5091:::newobject_2:head by client.4637.0:1 2024-09-17T18:10:49.343636+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:13:09.068 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 20 _merge_object_divergent_entries: keeping 45'63 (0'0) modify 1:75ac5091:::newobject_2:head by client.4637.0:1 2024-09-17T18:10:49.343636+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 45'63 last_divergent_update: 45'63 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.723+0000 7f8faab71640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 20 _merge_object_divergent_entries: keeping 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-17T18:13:09.069 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 45'60 last_divergent_update: 45'60 2024-09-17T18:13:09.077 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 45'60 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 45'60 (29'3) delete 1:8881e6d8:::existing_3:head by client.4635.0:1 2024-09-17T18:10:49.343084+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: false, new_object: false 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:12:38.983+0000 7f8fabb73640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'42 2024-09-17T18:13:09.078 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-17T18:13:09.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:13:09.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-17T18:13:09.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:13:09.118 INFO:tasks.workunit.client.0.smithi157.stdout:success 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.33174.out 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33174.out': No such file or directory 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-17T18:13:09.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-17T18:13:09.715 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' does not exist 2024-09-17T18:13:09.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-17T18:13:09.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:13:09.728 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:13:09.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:13:09.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:13:09.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:13:10.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:13:10.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:13:10.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:13:10.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:13:10.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:13:10.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:13:10.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:13:10.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:13:10.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:13:10.045 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:13:10.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:13:10.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:13:10.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:13:10.046 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:13:10.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:13:10.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:13:10.048 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:13:10.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:13:10.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:13:10.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:13:10.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:13:10.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:13:10.087 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:13:10.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:13:10.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:13:10.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:13:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:13:10.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:13:10.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:13:10.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:13:10.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:13:10.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:13:10.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:13:10.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:13:10.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:13:10.093 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:13:10.093 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.093 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:13:10.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:13:10.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:13:10.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-17T18:13:10.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:13:10.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.33174 2024-09-17T18:13:10.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:13:10.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-17T18:13:10.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-17T18:13:10.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-17T18:13:10.100 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-17T18:13:10.100 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-17T18:13:10.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:13:10.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-17T18:13:10.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:10.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:13:10.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:13:10.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:13:10.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:13:10.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:13:10.742 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.743 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.744 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:13:10.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:13:10.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get fsid 2024-09-17T18:13:10.824 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:13:10.825 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-mon.a.asok 2024-09-17T18:13:10.826 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:13:10.827 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.33174/ceph-mon.a.asok config get mon_host 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-17T18:13:10.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:13:11.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:13:11.763 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:11.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:13:11.765 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:13:11.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:13:11.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:13:11.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:13:11.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:13:11.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:13:11.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 70ffa155-c311-4dd8-bd3d-73ce6d572667' 2024-09-17T18:13:11.804 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:13:11.805 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:13:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3xulmGfEVMRAA/46187qONIDePMC3APH5hA== 2024-09-17T18:13:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3xulmGfEVMRAA/46187qONIDePMC3APH5hA=="}' 2024-09-17T18:13:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 70ffa155-c311-4dd8-bd3d-73ce6d572667 -i td/divergent-priors/0/new.json 2024-09-17T18:13:12.090 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:13:12.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-17T18:13:12.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3xulmGfEVMRAA/46187qONIDePMC3APH5hA== --osd-uuid 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:13:12.137 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:12.135+0000 7f2cf03a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:12.158 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:12.156+0000 7f2cf03a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:12.159 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:12.157+0000 7f2cf03a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:12.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:12.157+0000 7f2cf03a8640 -1 bdev(0x5585ef7d3c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:13:12.160 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:12.157+0000 7f2cf03a8640 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-17T18:13:17.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-17T18:13:17.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:13:17.803 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:13:17.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:13:17.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:13:18.342 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:13:18.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:13:18.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:13:18.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:13:18.342 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:13:18.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:13:18.377 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:18.374+0000 7fba35ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:18.379 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:18.376+0000 7fba35ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:18.379 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:18.377+0000 7fba35ad6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:18.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:13:18.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:13:18.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:13:18.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:13:18.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:13:18.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:18.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:13:18.793 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:13:18.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:18.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:19.216 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:19.214+0000 7fba35ad6640 -1 Falling back to public interface 2024-09-17T18:13:19.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:20.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:20.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:20.229 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:13:20.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:13:20.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:20.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:20.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:21.597 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:21.596+0000 7fba35ad6640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:13:21.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:21.662 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:13:21.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:21.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:13:21.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:21.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:22.182 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:23.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:23.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:23.184 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:13:23.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:13:23.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:23.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:23.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:24.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:24.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:24.626 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:13:24.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:13:24.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:24.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:25.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:26.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:26.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:26.037 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:13:26.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:13:26.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:26.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:26.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:27.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:27.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:27.466 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:13:27.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:13:27.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:27.467 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:27.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:28.879 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:28.880 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:13:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:13:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:28.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:29.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:30.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:30.304 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:13:30.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:30.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:13:30.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:30.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:30.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:31.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:31.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:31.732 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:13:31.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:13:31.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:31.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:32.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:33.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:33.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:33.163 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:13:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:13:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:33.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:33.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:34.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:34.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:34.934 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:13:34.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:13:34.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:34.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:35.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:36.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:36.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:36.438 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:13:36.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:13:36.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:36.439 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:37.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:38.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:38.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:38.106 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:13:38.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:13:38.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:38.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:39.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:40.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:40.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:40.144 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:13:40.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:13:40.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:40.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:40.861 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:41.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:41.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:41.863 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:13:41.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:13:41.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:41.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:42.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:43.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:43.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:43.631 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:13:43.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:13:43.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:43.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:13:44.091 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2906305640,v1:127.0.0.1:6803/2906305640] [v2:127.0.0.1:6804/2906305640,v1:127.0.0.1:6805/2906305640] exists,up 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:13:44.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:13:44.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:13:44.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:13:44.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-17T18:13:44.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:13:44.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:13:44.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:13:44.098 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:13:44.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0c83c4ef-86d6-4c59-9004-90b6db94f65e' 2024-09-17T18:13:44.099 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:13:44.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDYxulm7BMCBxAAN6mPZIggZ9x1Sd3Kj8JTXQ== 2024-09-17T18:13:44.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDYxulm7BMCBxAAN6mPZIggZ9x1Sd3Kj8JTXQ=="}' 2024-09-17T18:13:44.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c83c4ef-86d6-4c59-9004-90b6db94f65e -i td/divergent-priors/1/new.json 2024-09-17T18:13:44.586 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:13:44.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-17T18:13:44.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYxulm7BMCBxAAN6mPZIggZ9x1Sd3Kj8JTXQ== --osd-uuid 0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:13:44.634 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:44.632+0000 7fcbaf6f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:44.636 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:44.634+0000 7fcbaf6f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:44.635+0000 7fcbaf6f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:44.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:44.635+0000 7fcbaf6f1640 -1 bdev(0x55f8ed509c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:13:44.638 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:44.635+0000 7fcbaf6f1640 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-17T18:13:50.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-17T18:13:50.880 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:13:50.882 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:13:50.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:13:50.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:13:51.425 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:13:51.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:13:51.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:13:51.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:13:51.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:13:51.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:13:51.460 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:51.457+0000 7fbbfa558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:51.462 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:51.459+0000 7fbbfa558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:51.463 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:51.460+0000 7fbbfa558640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:13:51.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:13:51.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:13:51.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:13:51.883 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:51.884 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:52.343 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:52.803 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:52.801+0000 7fbbfa558640 -1 Falling back to public interface 2024-09-17T18:13:53.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:53.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:53.345 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:13:53.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:13:53.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:53.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:53.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:54.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:54.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:54.803 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:13:54.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:13:54.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:54.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:55.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:55.649 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:13:55.646+0000 7fbbfa558640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:13:56.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:56.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:56.207 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:13:56.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:13:56.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:56.208 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:56.665 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:57.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:57.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:57.667 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:13:57.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:13:57.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:57.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:58.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:13:59.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:13:59.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:13:59.278 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:13:59.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:13:59.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:13:59.279 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:13:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:00.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:00.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:00.834 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:14:00.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:14:00.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:00.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:01.241 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:02.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:02.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:02.243 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:14:02.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:14:02.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:02.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:02.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:03.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:03.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:03.688 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:14:03.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:14:03.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:03.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:04.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:05.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:05.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:05.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:14:05.262 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:14:05.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:05.263 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:05.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:06.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:06.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:06.745 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:14:06.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:14:06.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:06.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:07.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:08.171 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:08.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:08.172 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:14:08.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:14:08.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:08.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:10.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:10.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:10.158 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:14:10.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:14:10.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:10.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:11.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:12.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:12.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:12.203 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:14:12.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:14:12.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:12.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:12.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:13.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:13.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:13.770 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:14:13.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:14:13.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:13.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:14.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:15.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:15.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:15.206 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:14:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:14:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:15.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:15.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:16.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:16.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:16.909 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:14:16.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:14:16.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:16.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:17.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:18.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:18.643 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:14:18.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:14:18.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:18.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:14:19.108 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/631259066,v1:127.0.0.1:6811/631259066] [v2:127.0.0.1:6812/631259066,v1:127.0.0.1:6813/631259066] exists,up 0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:14:19.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:14:19.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:14:19.117 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:14:19.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:14:19.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:14:19.119 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:14:19.119 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ab667dc2-d230-4267-b488-4f703aefe017' 2024-09-17T18:14:19.120 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:14:19.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD7xulmPhgCCBAAUtNYTd3ZZv0nc5D2r9CIUw== 2024-09-17T18:14:19.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD7xulmPhgCCBAAUtNYTd3ZZv0nc5D2r9CIUw=="}' 2024-09-17T18:14:19.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab667dc2-d230-4267-b488-4f703aefe017 -i td/divergent-priors/2/new.json 2024-09-17T18:14:19.765 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:14:19.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-17T18:14:19.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7xulmPhgCCBAAUtNYTd3ZZv0nc5D2r9CIUw== --osd-uuid ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:14:19.813 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:19.812+0000 7fd0e4a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:19.824 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:19.822+0000 7fd0e4a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:19.825 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:19.823+0000 7fd0e4a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:19.825 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:19.824+0000 7fd0e4a22640 -1 bdev(0x55964d387c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:14:19.825 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:19.824+0000 7fd0e4a22640 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-17T18:14:25.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-17T18:14:25.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:14:25.259 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:14:25.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:14:25.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:14:25.809 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:14:25.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:14:25.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:14:25.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:14:25.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:14:25.811 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:14:25.845 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:25.843+0000 7fd8b700f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:25.847 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:25.845+0000 7fd8b700f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:25.848 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:25.847+0000 7fd8b700f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:14:26.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:14:26.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:14:26.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:14:26.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:14:26.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:14:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:14:26.259 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:14:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:26.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:26.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:27.214 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:27.212+0000 7fd8b700f640 -1 Falling back to public interface 2024-09-17T18:14:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:27.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:14:27.694 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:14:27.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:27.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:28.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:29.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:29.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:29.112 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:14:29.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:14:29.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:29.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:29.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:30.449 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:14:30.447+0000 7fd8b700f640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:14:30.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:30.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:30.547 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:14:30.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:14:30.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:30.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:31.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:32.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:32.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:32.018 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:14:32.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:14:32.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:32.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:32.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:33.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:33.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:33.542 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:14:33.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:14:33.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:33.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:34.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:35.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:35.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:35.005 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:14:35.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:14:35.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:35.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:35.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:36.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:36.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:36.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:14:36.448 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:14:36.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:36.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:36.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:37.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:37.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:37.903 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:14:37.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:14:37.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:37.904 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:38.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:39.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:39.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:39.331 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:14:39.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:14:39.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:39.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:39.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:40.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:40.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:40.745 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:14:40.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:14:40.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:40.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:41.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:42.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:42.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:42.146 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:14:42.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:14:42.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:42.147 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:42.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:43.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:43.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:43.841 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:14:43.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:14:43.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:43.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:44.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:45.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:45.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:45.402 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:14:45.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:14:45.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:45.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:46.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:47.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:47.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:47.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:14:47.222 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:14:47.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:47.223 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:48.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:49.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:49.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:49.334 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:14:49.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:14:49.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:49.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:50.050 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:51.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:51.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:51.052 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:14:51.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:14:51.052 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:51.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:51.987 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:52.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:52.988 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:52.989 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:14:52.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:14:52.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:52.990 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:53.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:14:54.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:14:54.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:14:54.795 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:14:54.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:14:54.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:14:54.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:14:55.193 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4039455310,v1:127.0.0.1:6819/4039455310] [v2:127.0.0.1:6820/4039455310,v1:127.0.0.1:6821/4039455310] exists,up ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:14:55.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:14:55.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:14:55.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:14:55.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-17T18:14:55.802 INFO:tasks.workunit.client.0.smithi157.stderr:noout is set 2024-09-17T18:14:55.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-17T18:14:56.447 INFO:tasks.workunit.client.0.smithi157.stderr:noin is set 2024-09-17T18:14:56.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-17T18:14:57.359 INFO:tasks.workunit.client.0.smithi157.stderr:nodown is set 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=test 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T18:14:57.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-17T18:14:58.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-17T18:14:58.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-17T18:14:58.865 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' already exists 2024-09-17T18:14:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:14:59.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-17T18:14:59.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-17T18:15:00.482 INFO:tasks.workunit.client.0.smithi157.stderr:set pool 1 allow_ec_overwrites to true 2024-09-17T18:15:00.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T18:15:00.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:15:00.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:15:00.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:15:00.494 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:15:00.495 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:15:00.495 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:15:00.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:15:00.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:15:00.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:15:00.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:15:00.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:15:00.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:15:00.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:15:00.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:15:00.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:00.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:15:01.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T18:15:01.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T18:15:01.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T18:15:01.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:01.123 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:15:01.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-17T18:15:01.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-17T18:15:01.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-09-17T18:15:01.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:01.245 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509449' 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T18:15:01.366 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:01.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:15:01.367 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T18:15:01.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:01.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T18:15:01.369 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836503 2024-09-17T18:15:01.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T18:15:01.369 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:01.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-17T18:15:01.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:15:02.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:15:02.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-17T18:15:03.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:03.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-17T18:15:03.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:03.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:15:03.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-17T18:15:03.228 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:03.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-17T18:15:03.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-17T18:15:03.229 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672974 2024-09-17T18:15:03.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:15:03.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-17T18:15:03.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:03.661 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-17T18:15:03.661 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:03.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:15:03.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-17T18:15:03.662 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-17T18:15:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-17T18:15:03.664 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509449 2024-09-17T18:15:03.664 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:15:04.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-17T18:15:04.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:15:04.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:04.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:15:04.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:15:05.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:15:05.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:15:05.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:05.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:15:05.538 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:05.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:15:06.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-17T18:15:06.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-17T18:15:06.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-17T18:15:06.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:06.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:15:06.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T18:15:06.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T18:15:06.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-09-17T18:15:06.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:06.211 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:15:06.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-17T18:15:06.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-17T18:15:06.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509452' 2024-09-17T18:15:06.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:06.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-17T18:15:06.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:06.330 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:15:06.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-17T18:15:06.330 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:06.331 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836506 2024-09-17T18:15:06.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-17T18:15:06.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-17T18:15:06.332 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:06.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-17T18:15:06.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:15:07.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:15:07.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:08.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-17T18:15:08.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:08.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T18:15:08.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:08.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:15:08.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T18:15:08.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:08.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T18:15:08.198 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T18:15:08.198 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672977 2024-09-17T18:15:08.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:15:08.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-17T18:15:08.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:08.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-17T18:15:08.630 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:08.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:15:08.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-17T18:15:08.631 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:08.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-17T18:15:08.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-17T18:15:08.632 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509452 2024-09-17T18:15:08.632 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:15:09.065 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:15:09.066 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:15:09.066 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:15:09.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:15:09.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:15:09.066 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:15:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:15:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:15:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:15:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:15:09.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:15:09.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:09.582 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:15:09.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-17T18:15:09.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-17T18:15:09.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-17T18:15:09.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:09.705 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:15:09.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-17T18:15:09.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-17T18:15:09.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672980' 2024-09-17T18:15:09.842 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:09.843 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-42949672980 2-64424509455' 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-17T18:15:09.993 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:09.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:15:09.994 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-17T18:15:09.994 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:09.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-17T18:15:09.995 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-17T18:15:09.996 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836509 2024-09-17T18:15:09.996 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:10.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-17T18:15:10.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:10.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-17T18:15:10.420 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:10.420 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:15:10.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-17T18:15:10.421 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:10.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-17T18:15:10.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-17T18:15:10.422 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672980 2024-09-17T18:15:10.423 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:15:10.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672980 2024-09-17T18:15:10.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:15:11.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:15:11.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:15:12.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-17T18:15:12.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:12.306 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T18:15:12.306 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:12.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:15:12.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T18:15:12.307 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:12.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:15:12.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T18:15:12.309 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509455 2024-09-17T18:15:12.309 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:15:12.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-17T18:15:12.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:15:12.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:12.740 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:13.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:15:13.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:15:13.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:15:13.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:15:13.269 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:15:13.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:15:13.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:15:13.270 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:15:13.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:15:13.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:15:13.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:13.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:14.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:15:14.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:15:14.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:15:14.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-17T18:15:14.201 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-17T18:15:14.605 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:15:14.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-17T18:15:14.617 INFO:tasks.workunit.client.0.smithi157.stdout:primary and soon to be divergent is 1 2024-09-17T18:15:14.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-17T18:15:14.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:15:14.992 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:15:14.992 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:14:59.845252+0000 0'0 29:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:08:00.075258+0000 0 0 2024-09-17T18:15:14.992 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:15:14.992 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:15:14.993 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:15:15.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-17T18:15:15.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:15.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-17T18:15:15.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-17T18:15:15.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-17T18:15:15.005 INFO:tasks.workunit.client.0.smithi157.stdout:writing initial objects 2024-09-17T18:15:15.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-17T18:15:15.006 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-17T18:15:15.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-17T18:15:15.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-17T18:15:15.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-17T18:15:15.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-17T18:15:15.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-17T18:15:15.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:15.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-17T18:15:16.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:16.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-17T18:15:16.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:16.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-17T18:15:16.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:16.613 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-17T18:15:16.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:16.770 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-17T18:15:17.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:17.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-17T18:15:17.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:17.348 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-17T18:15:17.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:17.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-17T18:15:17.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:17.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-17T18:15:17.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:17.994 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-17T18:15:18.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:18.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-17T18:15:18.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:18.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-17T18:15:18.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:18.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-17T18:15:18.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:18.852 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-17T18:15:19.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:19.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-17T18:15:19.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:19.294 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-17T18:15:19.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:19.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-17T18:15:19.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:19.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-17T18:15:19.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:19.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-17T18:15:20.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:20.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-17T18:15:20.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:20.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-17T18:15:20.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:20.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-17T18:15:20.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:20.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-17T18:15:20.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:20.905 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-17T18:15:21.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:21.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-17T18:15:21.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:21.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-17T18:15:21.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:21.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-17T18:15:21.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:21.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-17T18:15:21.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:21.860 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-17T18:15:22.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:22.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-17T18:15:22.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:22.357 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-17T18:15:22.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:22.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-17T18:15:22.725 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:22.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-17T18:15:22.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:22.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-17T18:15:23.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:23.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-17T18:15:23.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:23.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-17T18:15:23.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:23.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-17T18:15:23.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:23.632 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-17T18:15:23.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:23.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-17T18:15:24.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:24.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-17T18:15:24.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:24.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-17T18:15:24.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:24.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-17T18:15:24.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:24.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-17T18:15:24.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:24.882 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-17T18:15:25.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:25.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-17T18:15:25.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:25.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-17T18:15:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-17T18:15:25.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:25.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-17T18:15:25.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:25.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-17T18:15:26.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:26.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-17T18:15:26.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:26.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-17T18:15:26.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:26.635 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-17T18:15:26.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:26.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-17T18:15:27.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:27.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-17T18:15:27.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:27.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-17T18:15:27.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:27.407 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-17T18:15:27.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:27.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-17T18:15:27.809 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:27.810 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-17T18:15:28.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-17T18:15:28.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-17T18:15:28.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-17T18:15:28.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-17T18:15:28.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-17T18:15:28.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:28.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-17T18:15:29.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:29.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-17T18:15:29.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:29.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-17T18:15:29.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:29.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-17T18:15:29.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:29.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-17T18:15:29.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-17T18:15:30.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-17T18:15:30.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-17T18:15:30.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-17T18:15:30.779 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.780 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-17T18:15:30.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:30.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-17T18:15:31.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:31.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-17T18:15:31.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:31.334 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-17T18:15:31.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:31.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-17T18:15:31.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:31.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-17T18:15:31.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:31.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-17T18:15:32.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:32.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-17T18:15:32.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:32.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-17T18:15:32.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:32.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-17T18:15:32.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:32.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-17T18:15:32.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:32.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-17T18:15:33.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:33.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-17T18:15:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:33.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-17T18:15:33.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:33.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-17T18:15:33.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:33.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-17T18:15:34.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:34.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-17T18:15:34.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:34.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-17T18:15:34.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:34.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-17T18:15:34.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:34.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-17T18:15:34.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:34.773 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-17T18:15:35.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:35.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-17T18:15:35.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-17T18:15:35.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-17T18:15:35.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:15:35.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-17T18:15:35.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:15:35.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:15:35.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:15:35.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:15:35.413 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:15:35.413 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:15:35.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:15:35.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:15:35.414 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:15:35.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:15:35.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:15:35.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:15:35.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:15:35.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:15:35.466 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:15:35.862 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:15:35.863 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:15:35.863 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:15:35.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:15:35.863 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:35.863 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:15:35.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-17T18:15:35.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-17T18:15:35.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-17T18:15:35.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:35.985 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:15:36.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-17T18:15:36.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-17T18:15:36.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672987' 2024-09-17T18:15:36.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:15:36.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:15:36.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-17T18:15:36.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-17T18:15:36.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672987 2-64424509462' 2024-09-17T18:15:36.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:36.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-17T18:15:36.229 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:36.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:15:36.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-17T18:15:36.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:36.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-17T18:15:36.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-17T18:15:36.231 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 21474836516 2024-09-17T18:15:36.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:36.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836516 2024-09-17T18:15:36.666 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:15:37.667 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:15:37.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:15:38.080 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-17T18:15:38.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:38.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-17T18:15:38.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:38.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:15:38.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-17T18:15:38.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:38.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-17T18:15:38.083 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 42949672987 2024-09-17T18:15:38.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-17T18:15:38.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:15:38.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-17T18:15:38.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:15:38.503 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-17T18:15:38.503 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:15:38.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:15:38.504 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-17T18:15:38.504 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:15:38.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-17T18:15:38.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-17T18:15:38.505 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 64424509462 2024-09-17T18:15:38.506 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:15:38.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-17T18:15:38.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:15:38.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:38.938 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:39.468 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:15:39.469 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:15:39.878 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:15:39.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:15:39.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:15:39.879 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:15:40.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-17T18:15:40.413 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:15:40.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-17T18:15:40.859 INFO:tasks.workunit.client.0.smithi157.stdout:blackholing osds 0 2 2024-09-17T18:15:40.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-17T18:15:40.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:15:41.230 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:15:41.231 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-17T18:14:59.845252+0000 29'100 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T20:08:00.075258+0000 0 0 2024-09-17T18:15:41.231 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:15:41.231 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:15:41.231 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:15:41.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:15:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-17T18:15:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:15:41.242 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-17T18:15:41.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-17T18:15:41.320 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:15:41.320 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:15:41.320 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:15:41.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-17T18:15:41.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-17T18:15:41.405 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:15:41.406 INFO:tasks.workunit.client.0.smithi157.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-17T18:15:41.406 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:15:41.412 INFO:tasks.workunit.client.0.smithi157.stdout:writing divergent object 2024-09-17T18:15:41.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-17T18:15:41.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-17T18:15:41.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-17T18:15:42.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-17T18:15:42.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-17T18:15:42.692 INFO:tasks.workunit.client.0.smithi157.stdout:created pool test snap snap1 2024-09-17T18:15:42.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-17T18:15:42.696 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-17T18:15:42.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-17T18:15:52.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-17T18:15:52.700 INFO:tasks.workunit.client.0.smithi157.stderr:rados: no process found 2024-09-17T18:15:52.700 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-17T18:15:52.700 INFO:tasks.workunit.client.0.smithi157.stdout:killing all the osds 2024-09-17T18:15:52.701 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:15:53.083 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:15:53.083 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2024-09-17T18:14:59.845252+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:37:47.565718+0000 0 0 2024-09-17T18:15:53.083 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:15:53.083 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:15:53.084 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:15:53.094 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-17T18:15:53.095 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:15:53.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:15:53.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:15:53.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:15:53.096 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:15:53.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:15:53.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:53.201 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-17T18:15:53.697 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:15:53.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:53.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-17T18:15:54.215 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:15:54.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-17T18:15:54.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-17T18:15:54.733 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:15:54.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:15:54.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-17T18:15:55.244 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already out. 2024-09-17T18:15:55.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:15:55.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-17T18:15:55.750 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already out. 2024-09-17T18:15:55.762 INFO:tasks.workunit.client.0.smithi157.stdout:bringing up non_divergent 0 2 2024-09-17T18:15:55.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-17T18:15:55.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:15:56.140 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:15:56.140 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2024-09-17T18:14:59.845252+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-19T00:37:47.565718+0000 0 0 2024-09-17T18:15:56.140 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:15:56.140 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:15:56.140 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:15:56.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:15:56.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:15:56.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:15:56.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:15:56.157 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:15:56.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:15:56.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:15:56.158 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-17T18:15:56.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:15:56.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:15:56.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:15:56.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:15:56.189 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:15:56.187+0000 7f4dce20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:15:56.191 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:15:56.189+0000 7f4dce20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:15:56.192 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:15:56.190+0000 7f4dce20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:15:56.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:15:56.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:15:56.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:15:56.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:15:56.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:15:56.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:15:57.527 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:15:57.525+0000 7f4dce20f640 -1 Falling back to public interface 2024-09-17T18:15:57.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:15:57.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:15:57.982 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:15:57.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:15:57.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:15:57.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:15:58.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:15:59.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:15:59.383 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:15:59.383 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:15:59.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:15:59.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:15:59.384 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:15:59.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:16:00.697 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:00.694+0000 7f4dce20f640 -1 osd.0 29 log_to_monitors true 2024-09-17T18:16:00.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:16:00.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:00.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:16:00.791 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:16:00.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:00.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:16:01.253 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1007002176,v1:127.0.0.1:6803/1007002176] [v2:127.0.0.1:6804/1007002176,v1:127.0.0.1:6805/1007002176] exists,up 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:16:01.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:16:01.257 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:01.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:16:01.259 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:16:01.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:16:01.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-17T18:16:01.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:16:01.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:16:01.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:16:01.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:16:01.291 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:01.288+0000 7fef879d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:01.293 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:01.290+0000 7fef879d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:01.294 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:01.291+0000 7fef879d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:16:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:16:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:16:01.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:16:01.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:16:01.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:01.689 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:16:01.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:16:01.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:01.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:16:02.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:16:02.141 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:02.139+0000 7fef879d9640 -1 Falling back to public interface 2024-09-17T18:16:03.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:16:03.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:03.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:16:03.081 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:16:03.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:03.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:16:03.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:16:04.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:16:04.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:04.525 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:16:04.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:16:04.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:04.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:16:04.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:16:05.555 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:05.552+0000 7fef879d9640 -1 osd.2 29 log_to_monitors true 2024-09-17T18:16:05.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:16:05.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:05.927 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:16:05.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:16:05.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:05.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:16:06.367 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/569113160,v1:127.0.0.1:6811/569113160] [v2:127.0.0.1:6812/569113160,v1:127.0.0.1:6813/569113160] exists,up ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:16:06.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:16:06.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:16:06.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:16:06.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:16:06.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-17T18:16:06.916 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already in. 2024-09-17T18:16:06.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:16:06.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-17T18:16:07.484 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already in. 2024-09-17T18:16:07.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-17T18:16:12.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-17T18:16:12.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-17T18:16:12.498 INFO:tasks.workunit.client.0.smithi157.stdout:writing non-divergent object existing_4 2024-09-17T18:16:12.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-17T18:16:12.499 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:16:12.882 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:16:12.882 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-17T18:16:08.193748+0000 29'100 43:129 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-19T06:06:06.281149+0000 0 0 2024-09-17T18:16:12.882 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:16:12.882 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:16:12.882 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:16:12.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:16:13.033 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:16:13.034 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:16:13.034 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:16:13.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:16:13.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:16:13.035 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:16:13.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:13.520 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:16:13.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789957 2024-09-17T18:16:13.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789957 2024-09-17T18:16:13.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957' 2024-09-17T18:16:13.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:13.643 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:16:13.732 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:16:13.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:16:13.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:16:13.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:16:13.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:13.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:16:13.855 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724548 2024-09-17T18:16:13.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724548 2024-09-17T18:16:13.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-17T18:16:13.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:16:13.856 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789957 2024-09-17T18:16:13.856 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:16:13.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:16:13.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789957 2024-09-17T18:16:13.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:16:13.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789957 2024-09-17T18:16:13.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-17T18:16:13.859 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 158913789957 2024-09-17T18:16:13.859 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:16:14.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-17T18:16:14.314 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:16:14.315 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-167503724548 2024-09-17T18:16:14.315 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:16:14.315 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:16:14.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-167503724548 2024-09-17T18:16:14.316 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:16:14.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724548 2024-09-17T18:16:14.317 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 167503724548 2024-09-17T18:16:14.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-17T18:16:14.317 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:16:14.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724546 -lt 167503724548 2024-09-17T18:16:14.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:16:15.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:16:15.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:16:16.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724549 -lt 167503724548 2024-09-17T18:16:16.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:16:16.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:16.184 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:16.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:16:16.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:16.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:16.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:16.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:16.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:16.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:16.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:17.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:17.119 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:17.119 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:17.119 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:17.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:17.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:16:17.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:16:17.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:16:17.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:17.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:16:17.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:17.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:17.755 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:18.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:18.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:18.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:18.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:18.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:18.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:19.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:19.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:19.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:16:19.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:19.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:16:19.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:19.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:19.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:19.447 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:19.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:19.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:19.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:19.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:19.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:19.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:19.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:19.830 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:20.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:20.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:20.364 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:16:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:20.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:21.311 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:21.697 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:21.697 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:21.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:21.698 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:22.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:22.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:22.235 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:22.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:22.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:22.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:16:22.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:22.807 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:16:23.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:23.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:23.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:23.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:23.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:23.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:23.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:23.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:23.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:23.996 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:23.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:23.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:24.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:24.528 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:25.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:25.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:25.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:16:25.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:25.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:16:26.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:26.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:26.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:27.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:27.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:27.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:27.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:27.636 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:28.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:28.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:28.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:16:28.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:28.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:16:31.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:31.390 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:31.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:31.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:31.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:31.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:31.391 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:31.392 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:31.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:31.785 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:31.786 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:31.786 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:32.328 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:32.329 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:32.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:32.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:32.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:16:32.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:32.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:16:39.274 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:39.275 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:39.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:39.275 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:39.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:39.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:39.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:39.276 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:39.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:39.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:39.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:39.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:40.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:40.197 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:40.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:40.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:40.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:16:40.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:16:40.722 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:16:48.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:16:48.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:16:48.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:16:48.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:16:48.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:16:48.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:16:48.426 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:16:48.957 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:16:48.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:16:48.958 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:16:49.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:16:49.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:16:49.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:16:49.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:16:50.072 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:16:50.072 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "85682b4c-c606-46f7-90fd-2d88453db5b2", 2024-09-17T18:16:50.072 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:16:50.072 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:16:50.070189+0000", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:16:50.073 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.074 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.075 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.076 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:13:10.124059Z", 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:13:10.124059Z", 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:16:50.077 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:16:50.078 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.079 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:16:50.080 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 43, 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:13:10.993958+0000", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:16:07.882981+0000", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:16:05.757558+0000", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:16:07.277368+0000", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:16:50.081 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 10, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:16:50.082 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:14:58.542480+0000", 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 16389, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 3, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 1, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:16:50.083 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 1, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 1, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 1, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 1, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 1, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:16:50.084 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "30", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "pool", 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 1, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 30, 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [ 2024-09-17T18:16:50.085 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "snapid": 1, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "stamp": "2024-09-17T18:15:42.451673+0000", 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "snap1" 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:16:50.086 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "myprofile", 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:16:50.087 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 8192, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {} 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:16:50.088 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "70ffa155-c311-4dd8-bd3d-73ce6d572667", 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 5, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 37, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 41, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 31, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.089 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.090 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.091 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.092 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1007002176 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/1007002176", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/1007002176", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1007002176", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1007002176", 2024-09-17T18:16:50.093 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "0c83c4ef-86d6-4c59-9004-90b6db94f65e", 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:16:50.094 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 25, 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 32, 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.095 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.096 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.097 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.098 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/631259066", 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/631259066", 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/631259066", 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/631259066", 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "ab667dc2-d230-4267-b488-4f703aefe017", 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:16:50.099 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 15, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 29, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 39, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 0, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 33, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.100 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:16:50.101 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.102 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 569113160 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.103 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/569113160", 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/569113160", 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/569113160", 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/569113160", 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.104 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:15:53.529521+0000", 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:13:43.001855+0000", 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:15:54.046762+0000", 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:16:50.105 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:14:18.315686+0000", 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:15:54.557417+0000", 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:14:53.434587+0000", 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.106 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.107 INFO:tasks.workunit.client.0.smithi157.stdout: "myprofile": { 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-device-class": "", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-failure-domain": "osd", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-num-failure-domains": "0", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-osds-per-failure-domain": "0", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-root": "default", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "1", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van", 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "w": "8" 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.108 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.109 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/1007002176,v1:127.0.0.1:6805/1007002176]", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:16:50.110 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:13:17.165928Z", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:16:50.111 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/1007002176,v1:127.0.0.1:6803/1007002176]", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1007002176,v1:127.0.0.1:6809/1007002176]", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1007002176,v1:127.0.0.1:6807/1007002176]", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:16:50.112 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/631259066,v1:127.0.0.1:6813/631259066]", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:16:50.113 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:16:50.114 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:13:50.250449Z", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/631259066,v1:127.0.0.1:6811/631259066]", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/631259066,v1:127.0.0.1:6817/631259066]", 2024-09-17T18:16:50.115 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/631259066,v1:127.0.0.1:6815/631259066]", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:16:50.116 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/569113160,v1:127.0.0.1:6813/569113160]", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:16:50.117 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:14:24.615646Z", 2024-09-17T18:16:50.118 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/569113160,v1:127.0.0.1:6811/569113160]", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/569113160,v1:127.0.0.1:6817/569113160]", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/569113160,v1:127.0.0.1:6815/569113160]", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:16:50.119 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 0, 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:16:50.120 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [] 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 37 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 39 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 43, 2024-09-17T18:16:50.121 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:16:50.122 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:16:50.123 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.124 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:16:50.125 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:16:50.126 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:16:50.127 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.128 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.129 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:16:50.130 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:16:50.131 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:16:50.132 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.133 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 1, 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "test", 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 3, 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "set_chooseleaf_tries", 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 5 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "set_choose_tries", 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 100 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.134 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_indep", 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:16:50.135 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stderr:report 2111710746 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:16:50.136 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:16:50.137 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 1, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.138 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:16:50.140 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:16:50.140 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:13:10:969457+0000", 2024-09-17T18:16:50.140 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:16:50.140 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:16:50.140 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:16:50.141 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:16:50.142 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 1, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48454, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 101, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 1, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 303, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 101, 2024-09-17T18:16:50.143 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 101, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:16:50.144 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:16:50.145 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:16:50.146 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 18, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 18, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 3, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 3, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:16:50.147 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83696, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2164, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489104, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036842496, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2215936, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1550914, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:16:50.148 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10399, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416929 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:16:50.149 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:16:50.150 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 83696, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2164, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 10, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 81461, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314489104, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322036842496, 2024-09-17T18:16:50.151 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2215936, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1550914, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 10399, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 83416929 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:16:50.152 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:16:50.153 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48454, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 101, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 1, 2024-09-17T18:16:50.154 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 303, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 101, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 101, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:16:50.155 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:16:50.156 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:16:50.157 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1241088, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1241088, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:16:50.158 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 18, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 18, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 3, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 3, 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.159 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 158913789964 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672990 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 167503724555 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:16:50.160 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 1 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2147483647, 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:16:50.161 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 1, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:16:50.162 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.163 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 5, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:15:55.728418+0000", 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 99, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 399, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 547, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:16:50.164 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:16:50.165 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:16:50.165 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:16:50.166 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-17T18:16:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-17T18:16:51.515 INFO:tasks.workunit.client.0.smithi157.stderr:PG '1.0' not found 2024-09-17T18:16:51.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-17T18:16:51.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-17T18:16:51.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:16:51.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:16:51.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:16:51.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:16:51.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:16:51.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:16:51.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:16:51.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-17T18:16:51.796 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:16:51.797 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:16:51.832 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:51.829+0000 7f7cf629c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:51.834 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:51.831+0000 7f7cf629c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:51.835 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:51.833+0000 7f7cf629c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:16:52.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:52.245 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:16:52.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:16:52.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:52.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:16:52.678 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/1007002176,v1:127.0.0.1:6803/1007002176] [v2:127.0.0.1:6804/1007002176,v1:127.0.0.1:6805/1007002176] exists,up 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:16:52.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:16:52.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:16:52.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:16:52.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:16:52.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-17T18:16:52.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:16:52.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:16:52.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:16:52.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:16:52.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:16:52.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-17T18:16:52.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-17T18:16:53.175 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:53.173+0000 7f7cf629c640 -1 Falling back to public interface 2024-09-17T18:16:54.273 INFO:tasks.workunit.client.0.smithi157.stderr:PG '1.0' not found 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:16:54.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:16:54.546 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:16:54.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:16:54.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-17T18:16:54.548 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:16:54.548 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:16:54.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:16:54.584 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:54.582+0000 7fbdb3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:54.586 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:54.584+0000 7fbdb3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:54.588 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:54.585+0000 7fbdb3613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:16:54.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:16:54.985 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:16:54.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:16:54.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:16:55.451 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/569113160,v1:127.0.0.1:6811/569113160] [v2:127.0.0.1:6812/569113160,v1:127.0.0.1:6813/569113160] exists,up ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:16:55.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:16:55.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:16:55.452 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-17T18:16:55.453 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-17T18:16:55.686 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:55.684+0000 7fbdb3613640 -1 Falling back to public interface 2024-09-17T18:16:56.187 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:56.185+0000 7f7cf629c640 -1 osd.0 43 log_to_monitors true 2024-09-17T18:16:57.877 INFO:tasks.workunit.client.0.smithi157.stderr:PG '1.0' not found 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:16:58.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:16:58.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-17T18:16:58.143 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:16:58.143 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:16:58.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:16:58.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:16:58.144 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:16:58.206 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:16:58.259 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:16:58.257+0000 7fbdb3613640 -1 osd.2 43 log_to_monitors true 2024-09-17T18:16:58.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:16:58.623 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:16:58.623 INFO:tasks.workunit.client.0.smithi157.stderr:2' 2024-09-17T18:16:58.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:16:58.624 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:58.624 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:16:58.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-17T18:16:58.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-17T18:16:58.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619' 2024-09-17T18:16:58.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:58.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:16:58.934 INFO:tasks.workunit.client.0.smithi157.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-17T18:16:58.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-17T18:16:58.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-17T18:16:58.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-17T18:16:58.937 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:16:58.937 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-197568495619 2024-09-17T18:16:59.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:16:59.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:16:59.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-197568495619 2024-09-17T18:16:59.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:16:59.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-17T18:16:59.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-09-17T18:16:59.062 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 197568495619 2024-09-17T18:16:59.063 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:16:59.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 197568495619 2024-09-17T18:16:59.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:17:00.496 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:17:00.497 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:17:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-17T18:17:00.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:17:00.942 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-17T18:17:00.942 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:17:00.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:17:00.943 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-17T18:17:00.943 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:17:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-17T18:17:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-17T18:17:00.944 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 214748364803 2024-09-17T18:17:00.945 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:17:01.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-17T18:17:01.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:17:01.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:01.368 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:01.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:01.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:02.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:02.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:02.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:02.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:02.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:02.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:17:02.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:17:02.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:17:02.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:02.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:02.952 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:03.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:03.336 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:03.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:03.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:03.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:03.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:03.874 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:04.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:04.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:04.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-17T18:17:04.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:04.426 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:17:04.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:04.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:05.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:05.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:05.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:05.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:05.537 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:06.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:06.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:06.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-17T18:17:06.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:06.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T18:17:06.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:06.529 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:06.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:06.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:06.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:06.910 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:07.437 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:07.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:07.438 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:08.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:08.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:08.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-17T18:17:08.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:08.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-17T18:17:08.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:08.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:08.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:08.808 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:08.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:08.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:08.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:08.809 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:09.199 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:09.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:09.199 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:09.200 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:09.726 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:09.727 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:10.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:10.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:10.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-17T18:17:10.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:10.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-17T18:17:11.910 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:11.911 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:12.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:12.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:12.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:12.304 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:12.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:12.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:13.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:13.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:13.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-17T18:17:13.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:13.369 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-17T18:17:16.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:16.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:16.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:16.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:16.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:16.570 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:16.571 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:16.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:16.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:16.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:16.961 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:17.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:17.491 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-17T18:17:18.042 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:18.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-17T18:17:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:24.444 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:24.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:24.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:24.444 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:24.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:24.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:24.445 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:24.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:24.841 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:24.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:24.842 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:25.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:25.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:25.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:25.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:25.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:25.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:25.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:25.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:25.374 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-17T18:17:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:17:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:17:33.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:17:33.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:17:33.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:17:33.646 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:17:33.646 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:17:33.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:17:33.647 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:17:34.179 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:17:34.180 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:17:34.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:17:34.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:17:34.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-17T18:17:34.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-17T18:17:35.248 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:17:35.249 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_fingerprint": "85682b4c-c606-46f7-90fd-2d88453db5b2", 2024-09-17T18:17:35.249 INFO:tasks.workunit.client.0.smithi157.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:17:35.249 INFO:tasks.workunit.client.0.smithi157.stdout: "commit": "1889a165b5128787ec693450a3d1c5df5d6dc203", 2024-09-17T18:17:35.249 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-17T18:17:35.245969+0000", 2024-09-17T18:17:35.249 INFO:tasks.workunit.client.0.smithi157.stdout: "tag": "", 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "health": { 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "status": "HEALTH_WARN", 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "checks": { 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "OSDMAP_FLAGS": { 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:17:35.250 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 17 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [], 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.251 INFO:tasks.workunit.client.0.smithi157.stdout: "OSD_DOWN": { 2024-09-17T18:17:35.252 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:17:35.252 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:17:35.252 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 osds down", 2024-09-17T18:17:35.252 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:17:35.253 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.253 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:17:35.253 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.253 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "osd.1 (root=default,host=smithi157) is down" 2024-09-17T18:17:35.253 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "PG_DEGRADED": { 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:17:35.254 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.255 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: "severity": "HEALTH_WARN", 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: "summary": { 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-17T18:17:35.256 INFO:tasks.workunit.client.0.smithi157.stdout: "count": 1 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: "detail": [ 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "application not enabled on pool 'test'" 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.257 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "muted": false 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "mutes": [] 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_first_committed": 1, 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap_last_committed": 1, 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "monmap": { 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:13:10.124059Z", 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:13:10.124059Z", 2024-09-17T18:17:35.258 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release": 19, 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "min_mon_release_name": "squid", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "election_strategy": 1, 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "disallowed_leaders: ": "", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": false, 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "tiebreaker_mon": "", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_ranks: ": "", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "features": { 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "persistent": [ 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "kraken", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "luminous", 2024-09-17T18:17:35.259 INFO:tasks.workunit.client.0.smithi157.stdout: "mimic", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap-prune", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "nautilus", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "octopus", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "pacific", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "elector-pinging", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "quincy", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "reef", 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "squid" 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: "optional": [] 2024-09-17T18:17:35.260 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "mons": [ 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "rank": 0, 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "a", 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.261 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115", 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 0 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:7115/0", 2024-09-17T18:17:35.262 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: "priority": 0, 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 0, 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_location": "{}" 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.263 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.264 INFO:tasks.workunit.client.0.smithi157.stdout: "quorum": [ 2024-09-17T18:17:35.264 INFO:tasks.workunit.client.0.smithi157.stdout: 0 2024-09-17T18:17:35.264 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.264 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap": { 2024-09-17T18:17:35.264 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 52, 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "fsid": "8562ea5f-8024-4dbe-9e2c-3daab902ae42", 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "created": "2024-09-17T18:13:10.993958+0000", 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:16:59.206268+0000", 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "last_up_change": "2024-09-17T18:16:58.627476+0000", 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "last_in_change": "2024-09-17T18:16:07.277368+0000", 2024-09-17T18:17:35.265 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_num": 5800384, 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_set": [ 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "nodown", 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "noin", 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "noout", 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "pglog_hardlimit", 2024-09-17T18:17:35.266 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snapdirs", 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "recovery_deletes", 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "sortbitwise" 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_version": 12, 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "full_ratio": 0.99000000953674316, 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_snapshot": "", 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_max": 1, 2024-09-17T18:17:35.267 INFO:tasks.workunit.client.0.smithi157.stdout: "max_osd": 3, 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: "require_min_compat_client": "luminous", 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: "min_compat_client": "jewel", 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: "require_osd_release": "squid", 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: "allow_crimson": false, 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: "pools": [ 2024-09-17T18:17:35.268 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_name": "test", 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "create_time": "2024-09-17T18:14:58.542480+0000", 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "flags": 16389, 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-17T18:17:35.269 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 3, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "size": 3, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "min_size": 2, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_rule": 1, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_count": 0, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_target": 0, 2024-09-17T18:17:35.270 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "is_stretch_pool": false, 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "object_hash": 2, 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_autoscale_mode": "off", 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num": 1, 2024-09-17T18:17:35.271 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num": 1, 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_placement_num_target": 1, 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_target": 1, 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_num_pending": 1, 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pg_merge_meta": { 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "source_pgid": "0.0", 2024-09-17T18:17:35.272 INFO:tasks.workunit.client.0.smithi157.stdout: "ready_epoch": 0, 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_started": 0, 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": 0, 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "source_version": "0'0", 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "target_version": "0'0" 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "last_change": "30", 2024-09-17T18:17:35.273 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend": "0", 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "auid": 0, 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_mode": "pool", 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_seq": 1, 2024-09-17T18:17:35.274 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_epoch": 30, 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_snaps": [ 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: "snapid": 1, 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: "stamp": "2024-09-17T18:15:42.451673+0000", 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "snap1" 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.275 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps": "[]", 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_bytes": 0, 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "quota_max_objects": 0, 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "tiers": [], 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "tier_of": -1, 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "read_tier": -1, 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "write_tier": -1, 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_mode": "none", 2024-09-17T18:17:35.276 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_bytes": 0, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "target_max_objects": 0, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_flush_age": 0, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "cache_min_evict_age": 0, 2024-09-17T18:17:35.277 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profile": "myprofile", 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_params": { 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "none" 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_period": 0, 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_count": 0, 2024-09-17T18:17:35.278 INFO:tasks.workunit.client.0.smithi157.stdout: "use_gmt_hitset": true, 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "min_read_recency_for_promote": 0, 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "min_write_recency_for_promote": 0, 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "hit_set_search_last_n": 0, 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "grade_table": [], 2024-09-17T18:17:35.279 INFO:tasks.workunit.client.0.smithi157.stdout: "stripe_width": 8192, 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: "expected_num_objects": 0, 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: "fast_read": false, 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: "options": {}, 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: "application_metadata": {} 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.280 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: "osds": [ 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "70ffa155-c311-4dd8-bd3d-73ce6d572667", 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:17:35.281 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 37, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 43, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 46, 2024-09-17T18:17:35.282 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 51, 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 45, 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.283 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6802", 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6803", 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.284 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6804", 2024-09-17T18:17:35.285 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6805", 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.286 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6808", 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.287 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6809", 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.288 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.289 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:17:35.289 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.289 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.289 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.289 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6806", 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6807", 2024-09-17T18:17:35.290 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1954171348 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6803/1954171348", 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6805/1954171348", 2024-09-17T18:17:35.291 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1954171348", 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1954171348", 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.292 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "0c83c4ef-86d6-4c59-9004-90b6db94f65e", 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 0, 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:17:35.293 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 0, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 0, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 10, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 25, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 32, 2024-09-17T18:17:35.294 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:17:35.295 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.296 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.297 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:17:35.298 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.299 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.300 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:17:35.301 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.302 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 631259066 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.303 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/631259066", 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/631259066", 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/631259066", 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/631259066", 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: "exists" 2024-09-17T18:17:35.304 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: "uuid": "ab667dc2-d230-4267-b488-4f703aefe017", 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 1, 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: "in": 1, 2024-09-17T18:17:35.305 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 1, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_affinity": 1, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_begin": 39, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "last_clean_end": 43, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 50, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "up_thru": 0, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "down_at": 49, 2024-09-17T18:17:35.306 INFO:tasks.workunit.client.0.smithi157.stdout: "lost_at": 0, 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addrs": { 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6810", 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.307 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6811", 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.308 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addrs": { 2024-09-17T18:17:35.309 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.309 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.309 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.309 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6812", 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6813", 2024-09-17T18:17:35.310 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addrs": { 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.311 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6816", 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.312 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6817", 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.313 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addrs": { 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: "addrvec": [ 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v2", 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6814", 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.314 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "v1", 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: "addr": "127.0.0.1:6815", 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: "nonce": 1531832634 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.315 INFO:tasks.workunit.client.0.smithi157.stdout: "public_addr": "127.0.0.1:6811/1531832634", 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "cluster_addr": "127.0.0.1:6813/1531832634", 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1531832634", 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1531832634", 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "state": [ 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "exists", 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: "up" 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.316 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_xinfo": [ 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:16:56.468254+0000", 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:17:35.317 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:13:43.001855+0000", 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:15:54.046762+0000", 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:17:35.318 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:14:18.315686+0000", 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:17:35.319 INFO:tasks.workunit.client.0.smithi157.stdout: "down_stamp": "2024-09-17T18:16:58.478767+0000", 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_probability": 0, 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "laggy_interval": 0, 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "features": 4540719139924082687, 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "old_weight": 0, 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "last_purged_snaps_scrub": "2024-09-17T18:14:53.434587+0000", 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: "dead_epoch": 0 2024-09-17T18:17:35.320 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap": [], 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_items": [], 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_upmap_primaries": [], 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: "pg_temp": [], 2024-09-17T18:17:35.321 INFO:tasks.workunit.client.0.smithi157.stdout: "primary_temp": [], 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "blocklist": {}, 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "range_blocklist": {}, 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "erasure_code_profiles": { 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "default": { 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "2", 2024-09-17T18:17:35.322 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van" 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "myprofile": { 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-device-class": "", 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-failure-domain": "osd", 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-num-failure-domains": "0", 2024-09-17T18:17:35.323 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-osds-per-failure-domain": "0", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "crush-root": "default", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "k": "2", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "m": "1", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "plugin": "jerasure", 2024-09-17T18:17:35.324 INFO:tasks.workunit.client.0.smithi157.stdout: "technique": "reed_sol_van", 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: "w": "8" 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: "removed_snaps_queue": [], 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: "new_removed_snaps": [], 2024-09-17T18:17:35.325 INFO:tasks.workunit.client.0.smithi157.stdout: "new_purged_snaps": [], 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "crush_node_flags": {}, 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "device_class_flags": {}, 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode": { 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_enabled": false, 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_bucket_count": 0, 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "degraded_stretch_mode": 0, 2024-09-17T18:17:35.326 INFO:tasks.workunit.client.0.smithi157.stdout: "recovering_stretch_mode": 0, 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: "stretch_mode_bucket": 0 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_metadata": [ 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:17:35.327 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6804/1954171348,v1:127.0.0.1:6805/1954171348]", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:17:35.328 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:17:35.329 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:13:17.165928Z", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:17:35.330 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6802/1954171348,v1:127.0.0.1:6803/1954171348]", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1954171348,v1:127.0.0.1:6809/1954171348]", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1954171348,v1:127.0.0.1:6807/1954171348]", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:17:35.331 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/0", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.332 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/631259066,v1:127.0.0.1:6813/631259066]", 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:17:35.333 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:17:35.334 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:13:50.250449Z", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/631259066,v1:127.0.0.1:6811/631259066]", 2024-09-17T18:17:35.335 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/631259066,v1:127.0.0.1:6817/631259066]", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/631259066,v1:127.0.0.1:6815/631259066]", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/1", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:17:35.336 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "arch": "x86_64", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "back_addr": "[v2:127.0.0.1:6812/1531832634,v1:127.0.0.1:6813/1531832634]", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "back_iface": "", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs": "1", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_db": "0", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_dedicated_wal": "0", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluefs_single_shared_device": "1", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_allocation_from_file": "0", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_access_mode": "file", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_block_size": "4096", 2024-09-17T18:17:35.337 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_devices": "sda", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_rotational": "1", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_size": "107374182400", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_support_discard": "1", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_bdev_type": "hdd", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "bluestore_min_alloc_size": "4096", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_release": "squid", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_short": "19.3.0-4975-g1889a165", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "ceph_version_when_created": "ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev)", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-17T18:17:35.338 INFO:tasks.workunit.client.0.smithi157.stdout: "created_at": "2024-09-17T18:14:24.615646Z", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "default_device_class": "hdd", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XFPK", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": "sda", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "distro": "centos", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_description": "CentOS Stream 9", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "distro_version": "9", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "front_addr": "[v2:127.0.0.1:6810/1531832634,v1:127.0.0.1:6811/1531832634]", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "front_iface": "", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1531832634,v1:127.0.0.1:6817/1531832634]", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1531832634,v1:127.0.0.1:6815/1531832634]", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "hostname": "smithi157", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "journal_rotational": "1", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024", 2024-09-17T18:17:35.339 INFO:tasks.workunit.client.0.smithi157.stdout: "kernel_version": "5.14.0-508.el9.x86_64", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_swap_kb": "0", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "mem_total_kb": "32487304", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "os": "Linux", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_data": "td/divergent-priors/2", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_objectstore": "bluestore", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "osdspec_affinity": "", 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "rotational": "1" 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_clean_epochs": { 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "min_last_epoch_clean": 26, 2024-09-17T18:17:35.340 INFO:tasks.workunit.client.0.smithi157.stdout: "last_epoch_clean": { 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "per_pool": [ 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "floor": 26 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_epochs": [ 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 46 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:17:35.341 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 50 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_first_committed": 1, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "osdmap_last_committed": 52, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "crushmap": { 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "devices": [ 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.0", 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:17:35.342 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.1", 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd.2", 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "class": "hdd" 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "types": [ 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 0, 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "osd" 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.343 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "host" 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 2, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "chassis" 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 3, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "rack" 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 4, 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "row" 2024-09-17T18:17:35.344 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 5, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pdu" 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 6, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "pod" 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 7, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "room" 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 8, 2024-09-17T18:17:35.345 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "datacenter" 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 9, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "zone" 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 10, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "region" 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "root" 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.346 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "buckets": [ 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -1, 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default", 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:17:35.347 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -2, 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "default~hdd", 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 11, 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "root", 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:17:35.348 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -3, 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157", 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.349 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.350 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "id": -4, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "name": "smithi157~hdd", 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "type_id": 1, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "type_name": "host", 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 19206, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "alg": "straw2", 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "hash": "rjenkins1", 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "items": [ 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 0, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 0 2024-09-17T18:17:35.351 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 1, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 1 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "id": 2, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "weight": 6402, 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: "pos": 2 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.352 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "rules": [ 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 0, 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "replicated_rule", 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 1, 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.353 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_firstn", 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_id": 1, 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "rule_name": "test", 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "type": 3, 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: "steps": [ 2024-09-17T18:17:35.354 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "set_chooseleaf_tries", 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 5 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "set_choose_tries", 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 100 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "take", 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "item": -1, 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "item_name": "default" 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.355 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "choose_indep", 2024-09-17T18:17:35.356 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 0, 2024-09-17T18:17:35.356 INFO:tasks.workunit.client.0.smithi157.stdout: "type": "osd" 2024-09-17T18:17:35.356 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.356 INFO:tasks.workunit.client.0.smithi157.stderr:report 1745779899 2024-09-17T18:17:35.356 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "op": "emit" 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: ] 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "tunables": { 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_tries": 0, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_local_fallback_tries": 0, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_total_tries": 50, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_descend_once": 1, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_vary_r": 1, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "chooseleaf_stable": 1, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_descents": 100, 2024-09-17T18:17:35.357 INFO:tasks.workunit.client.0.smithi157.stdout: "msr_collision_tries": 100, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "straw_calc_version": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "allowed_bucket_algs": 54, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "profile": "jewel", 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "optimal_tunables": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "legacy_tunables": 0, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "minimum_required_version": "jewel", 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables2": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v2_rules": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables3": 1, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v3_rules": 0, 2024-09-17T18:17:35.358 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v4_buckets": 1, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "require_feature_tunables5": 1, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "has_v5_rules": 0, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "has_msr_rules": 0 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "choose_args": {} 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "fsmap": { 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 1, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "btime": "2024-09-17T18:13:10:969457+0000", 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "default_fscid": -1, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": { 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "compat": {}, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "ro_compat": {}, 2024-09-17T18:17:35.359 INFO:tasks.workunit.client.0.smithi157.stdout: "incompat": { 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_1": "base v0.20", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_2": "client writeable ranges", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_3": "default file layouts on dirs", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_4": "dir inode in separate object", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_5": "mds uses versioned encoding", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_8": "no anchor table", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_9": "file layout v2", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_10": "snaprealm v2", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_11": "minor log segments", 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_12": "quiesce subvolumes" 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.360 INFO:tasks.workunit.client.0.smithi157.stdout: "feature_flags": { 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "enable_multiple": true, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "ever_enabled_multiple": true 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "standbys": [], 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "filesystems": [] 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_first_committed": 1, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "mdsmap_last_committed": 1, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "auth": { 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 5, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "num_secrets": 3 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.361 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_active": 1, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_unknown": 0, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osd": 3, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_sum": { 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48454, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 101, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 1, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 303, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 101, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:17:35.362 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 101, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:17:35.363 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:17:35.364 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.365 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 18, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 18, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 0 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum": { 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 85136, 2024-09-17T18:17:35.366 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2308, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 15, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 82736, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314487664, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322035367936, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2363392, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1612098, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 16266, 2024-09-17T18:17:35.367 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 84721782 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:17:35.368 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_sum_by_class": { 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "hdd": { 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "up_from": 0, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 0, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pgs": 3, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "num_osds": 3, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_osds": 3, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "num_per_pool_omap_osds": 3, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "kb": 314572800, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used": 85136, 2024-09-17T18:17:35.369 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_data": 2308, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_omap": 15, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_used_meta": 82736, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "kb_avail": 314487664, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "statfs": { 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 322122547200, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 322035367936, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 2363392, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1612098, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 16266, 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 84721782 2024-09-17T18:17:35.370 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "hb_peers": [], 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "snap_trim_queue_len": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "num_snap_trimming": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shards_repaired": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "op_queue_age_hist": { 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "histogram": [], 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "upper_bound": 1 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "perf_stat": { 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ms": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ms": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "commit_latency_ns": 0, 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: "apply_latency_ns": 0 2024-09-17T18:17:35.371 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "alerts": [], 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "network_ping_times": [] 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "pool_stats": [ 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "poolid": 1, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg": 1, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "stat_sum": { 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes": 48454, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects": 101, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_clones": 1, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_object_copies": 303, 2024-09-17T18:17:35.372 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing_on_primary": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_missing": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_degraded": 101, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_misplaced": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_unfound": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_dirty": 101, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_whiteouts": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_read_kb": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write": 101, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_write_kb": 101, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_scrub_errors": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_shallow_scrub_errors": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_deep_scrub_errors": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_recovered": 0, 2024-09-17T18:17:35.373 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_recovered": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_keys_recovered": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_omap": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_hit_set_archive": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_bytes_hit_set_archive": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_kb": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_kb": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_promote": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_high": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_flush_mode_low": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_some": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_evict_mode_full": 0, 2024-09-17T18:17:35.374 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_pinned": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_legacy_snapsets": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_large_omap_objects": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_manifest": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_bytes": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_omap_keys": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "num_objects_repaired": 0 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "store_stats": { 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "total": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "available": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "internally_reserved": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "allocated": 1241088, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "data_stored": 1241088, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed": 0, 2024-09-17T18:17:35.375 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_allocated": 0, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "data_compressed_original": 0, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "omap_allocated": 0, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "internal_metadata": 0 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "log_size": 18, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "ondisk_log_size": 18, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "up": 3, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "acting": 3, 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "num_store_stats": 3 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "osd_stats": [ 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.376 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 197568495626 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 42949672990 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "seq": 214748364810 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_state": [ 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "state": "active+undersized+degraded", 2024-09-17T18:17:35.377 INFO:tasks.workunit.client.0.smithi157.stdout: "num": 1 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_pg_by_osd": [ 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2147483647, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 2, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:17:35.378 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 0, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 1, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 1, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "osd": 1, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_primary_pg": 0, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_acting_pg": 0, 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "num_up_not_acting_pg": 0 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.379 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": [ 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: { 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "pool": 1, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "purged_snaps": {} 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: ], 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "servicemap": { 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "epoch": 5, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "modified": "2024-09-17T18:15:55.728418+0000", 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "services": {} 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: }, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_first_committed": 1, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "mgrstat_last_committed": 121, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_first_committed": 1, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "logm_last_committed": 481, 2024-09-17T18:17:35.380 INFO:tasks.workunit.client.0.smithi157.stdout: "paxos": { 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout: "first_committed": 1, 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout: "last_committed": 660, 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout: "last_pn": 0, 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout: "accepted_pn": 0 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout: } 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-17T18:17:35.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-17T18:17:35.382 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery 2024-09-17T18:17:35.382 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:35.647 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:35.647 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 52:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:35.647 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:35.647 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:35.648 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:35.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.0.asok 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-17T18:17:35.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.0.asok set_recovery_delay 100000 2024-09-17T18:17:35.743 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:17:35.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.2.asok 2024-09-17T18:17:35.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-17T18:17:35.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.2.asok set_recovery_delay 100000 2024-09-17T18:17:35.833 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:17:35.840 INFO:tasks.workunit.client.0.smithi157.stdout:revive divergent 1 2024-09-17T18:17:35.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-17T18:17:35.841 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:36.212 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:36.212 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 52:143 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:36.212 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:36.212 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:36.212 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:36.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-17T18:17:36.855 INFO:tasks.workunit.client.0.smithi157.stderr:noup is set 2024-09-17T18:17:36.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-17T18:17:36.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:17:36.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:17:36.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:17:36.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:17:36.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:17:36.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:17:36.875 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:17:36.876 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:17:36.905 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:36.903+0000 7f22c97bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:36.914 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:36.912+0000 7f22c97bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:36.915 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:36.913+0000 7f22c97bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:37.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-17T18:17:37.329 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-17T18:17:38.250 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:38.248+0000 7f22c97bf640 -1 Falling back to public interface 2024-09-17T18:17:40.819 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:40.815+0000 7f22c97bf640 -1 osd.1 30 log_to_monitors true 2024-09-17T18:17:42.331 INFO:tasks.workunit.client.0.smithi157.stdout:delay recovery divergent 2024-09-17T18:17:42.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-17T18:17:42.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:42.716 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:42.716 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:42.716 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:42.716 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:42.717 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.33174/ceph-osd.1.asok 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-17T18:17:42.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33174/ceph-osd.1.asok set_recovery_delay 100000 2024-09-17T18:17:42.810 INFO:tasks.workunit.client.0.smithi157.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-17T18:17:42.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-17T18:17:43.863 INFO:tasks.workunit.client.0.smithi157.stderr:noup is unset 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:17:43.874 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:17:43.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:17:43.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:17:44.318 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 46 up_thru 51 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/1954171348,v1:127.0.0.1:6803/1954171348] [v2:127.0.0.1:6804/1954171348,v1:127.0.0.1:6805/1954171348] exists,up 70ffa155-c311-4dd8-bd3d-73ce6d572667 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:17:44.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:17:44.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:17:44.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:17:44.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:17:44.320 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:17:44.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:17:44.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:17:44.321 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:17:44.736 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/3031951446,v1:127.0.0.1:6819/3031951446] [v2:127.0.0.1:6820/3031951446,v1:127.0.0.1:6821/3031951446] exists,up 0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:17:44.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:17:44.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:17:44.738 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:17:44.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:17:44.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:17:45.124 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/1531832634,v1:127.0.0.1:6811/1531832634] [v2:127.0.0.1:6812/1531832634,v1:127.0.0.1:6813/1531832634] exists,up ab667dc2-d230-4267-b488-4f703aefe017 2024-09-17T18:17:45.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:17:45.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:17:45.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:17:45.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:45.494 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:45.494 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:45.494 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:45.495 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:45.495 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:45.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-17T18:17:45.505 INFO:tasks.workunit.client.0.smithi157.stdout:wait for peering 2024-09-17T18:17:45.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:45.876 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:45.877 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:45.877 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:45.877 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:45.877 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:45.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-17T18:17:46.020 INFO:tasks.workunit.client.0.smithi157.stdout:killing divergent 1 2024-09-17T18:17:46.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-17T18:17:46.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:46.390 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:46.391 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:46.391 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:46.391 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:46.391 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:46.402 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-17T18:17:46.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:17:46.402 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:17:46.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:17:46.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:17:46.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:17:46.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:17:46.508 INFO:tasks.workunit.client.0.smithi157.stdout:reviving divergent 1 2024-09-17T18:17:46.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-17T18:17:46.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:17:46.879 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:17:46.879 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-17T18:16:59.516792+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T22:24:53.083782+0000 0 0 2024-09-17T18:17:46.879 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:17:46.879 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:17:46.880 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:17:46.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-17T18:17:46.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-17T18:17:46.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-17T18:17:46.894 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:17:46.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:17:46.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:17:46.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-17T18:17:46.896 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=8562ea5f-8024-4dbe-9e2c-3daab902ae42 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33174/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:17:46.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:17:46.929 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:46.926+0000 7fa09281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:46.931 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:46.928+0000 7fa09281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:46.932 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:46.930+0000 7fa09281f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:17:47.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:17:47.351 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:17:47.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:17:47.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:17:47.784 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/3031951446,v1:127.0.0.1:6819/3031951446] [v2:127.0.0.1:6820/3031951446,v1:127.0.0.1:6821/3031951446] exists,up 0c83c4ef-86d6-4c59-9004-90b6db94f65e 2024-09-17T18:17:47.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:17:47.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:17:47.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:17:47.785 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-17T18:17:48.288 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:48.286+0000 7fa09281f640 -1 Falling back to public interface 2024-09-17T18:17:51.213 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:17:51.211+0000 7fa09281f640 -1 osd.1 58 log_to_monitors true 2024-09-17T18:18:07.786 INFO:tasks.workunit.client.0.smithi157.stdout:allowing recovery 2024-09-17T18:18:07.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-17T18:18:07.787 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:18:08.171 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:18:08.171 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-17T18:17:52.826029+0000 58'103 62:160 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:49:01.875852+0000 0 0 2024-09-17T18:18:08.171 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:18:08.171 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:18:08.172 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:18:08.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-17T18:18:08.183 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-17T18:18:08.295 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:18:08.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-17T18:18:08.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-17T18:18:08.416 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:18:08.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-17T18:18:08.425 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-17T18:18:08.536 INFO:tasks.workunit.client.0.smithi157.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-17T18:18:08.545 INFO:tasks.workunit.client.0.smithi157.stdout:reading divergent objects 2024-09-17T18:18:08.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-17T18:18:08.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-17T18:18:08.917 INFO:tasks.workunit.client.0.smithi157.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:18:08.917 INFO:tasks.workunit.client.0.smithi157.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-17T18:17:52.826029+0000 58'103 62:160 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:14:58.612782+0000 0'0 2024-09-17T18:14:58.612782+0000 0 0 periodic scrub scheduled @ 2024-09-18T23:49:01.875852+0000 0 0 2024-09-17T18:18:08.918 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-17T18:18:08.918 INFO:tasks.workunit.client.0.smithi157.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:18:08.918 INFO:tasks.workunit.client.0.smithi157.stderr:dumped pgs 2024-09-17T18:18:08.930 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-17T18:18:08.930 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-17T18:18:08.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:18:08.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-17T18:18:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:18:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-17T18:18:09.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:18:09.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-17T18:18:09.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-17T18:18:09.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-17T18:18:09.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-17T18:18:09.101 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-17T18:18:09.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:18:09.107 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-17T18:15:35.032307+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-17T18:15:35.032307+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-17T18:15:42.733472+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-17T18:15:42.733472+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:18:09.108 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-17T18:15:42.733472+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-17T18:15:41.446080+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-17T18:15:42.452686+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.109 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-17T18:15:41.446080+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-17T18:15:42.452686+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-17T18:15:42.452686+0000 0 ObjectCleanRegions clean_offsets: [(483, 18446744073709551132)], clean_omap: true, new_object: false 2024-09-17T18:18:09.110 INFO:tasks.workunit.client.0.smithi157.stdout:td/divergent-priors/osd.1.log:2024-09-17T18:17:43.859+0000 7f22ad731640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-17T18:18:09.111 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-17T18:18:09.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-17T18:18:09.117 INFO:tasks.workunit.client.0.smithi157.stdout:success 2024-09-17T18:18:09.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-17T18:18:09.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-17T18:18:09.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-17T18:18:09.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-17T18:18:09.598 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'test' does not exist 2024-09-17T18:18:09.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-17T18:18:09.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:18:09.610 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:18:09.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:18:09.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:18:09.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:18:14.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:18:14.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-17T18:18:14.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:18:14.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:18:14.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:18:14.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:18:14.925 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:18:14.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:18:14.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:18:14.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:18:14.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:18:14.931 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:18:14.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:18:14.932 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:18:14.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:18:14.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:18:14.934 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:18:14.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:14.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:18:14.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:18:14.935 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:14.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:18:14.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:18:14.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:18:14.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:18:14.964 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:14.965 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:18:14.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/divergent-priors 0 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:18:14.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-17T18:18:14.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:18:14.967 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:18:14.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:18:14.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:18:14.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:18:14.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:18:14.968 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:18:14.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:18:14.970 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:18:14.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:18:14.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:18:14.971 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:18:14.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:14.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:18:14.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:18:14.973 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:14.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:18:14.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:18:14.974 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-17T18:18:14.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:18:14.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:14.976 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.33174 2024-09-17T18:18:14.976 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.33174 2024-09-17T18:18:14.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:18:14.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:18:14.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:18:14.978 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:18:14.978 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:18:15.008 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-17T18:18:15.009 DEBUG:teuthology.orchestra.run.smithi157:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-17T18:18:15.071 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:18:15.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ec-error-rollforward 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none ' 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-17T18:18:15.078 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-17T18:18:15.079 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ec-error-rollforward 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:18:15.081 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:18:15.082 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:18:15.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:18:15.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:18:15.083 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:18:15.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:15.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:18:15.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:18:15.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:18:15.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:18:15.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:18:15.086 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-17T18:18:15.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:18:15.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:15.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:15.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90642 2024-09-17T18:18:15.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:18:15.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:18:15.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ec-error-rollforward 2024-09-17T18:18:15.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:18:15.089 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:15.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:15.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.90642 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:18:15.092 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:18:15.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:18:15.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ec-error-rollforward/a 2024-09-17T18:18:15.093 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:15.579 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:18:15.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:18:15.764 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.90642/ceph-mon.a.asok 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:18:15.765 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.90642/ceph-mon.a.asok config get fsid 2024-09-17T18:18:15.844 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:18:15.844 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.90642/ceph-mon.a.asok 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:18:15.845 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.90642/ceph-mon.a.asok config get mon_host 2024-09-17T18:18:15.923 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ec-error-rollforward 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ec-error-rollforward/x 2024-09-17T18:18:15.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:16.761 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:18:16.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:18:16.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:16.792 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:18:16.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5848a82a-73b9-45ee-b4f9-de2d36a8a99e 2024-09-17T18:18:16.794 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 5848a82a-73b9-45ee-b4f9-de2d36a8a99e 2024-09-17T18:18:16.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5848a82a-73b9-45ee-b4f9-de2d36a8a99e' 2024-09-17T18:18:16.795 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:18:16.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDox+lmy7oEMBAAfHrf5EaDNWffJMqrp5T8jg== 2024-09-17T18:18:16.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDox+lmy7oEMBAAfHrf5EaDNWffJMqrp5T8jg=="}' 2024-09-17T18:18:16.808 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5848a82a-73b9-45ee-b4f9-de2d36a8a99e -i td/ec-error-rollforward/0/new.json 2024-09-17T18:18:17.079 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:18:17.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-17T18:18:17.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDox+lmy7oEMBAAfHrf5EaDNWffJMqrp5T8jg== --osd-uuid 5848a82a-73b9-45ee-b4f9-de2d36a8a99e 2024-09-17T18:18:17.116 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:17.115+0000 7efca2a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:17.148 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:17.146+0000 7efca2a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:17.149 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:17.148+0000 7efca2a2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:17.149 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:17.148+0000 7efca2a2c640 -1 bdev(0x55d62e5f7c00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:17.149 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:17.148+0000 7efca2a2c640 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-17T18:18:22.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-17T18:18:22.726 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:18:22.727 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:18:22.727 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:18:22.728 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:18:23.269 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:18:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:18:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:18:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:18:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:18:23.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:18:23.294 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:23.293+0000 7f6ade013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:23.310 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:23.309+0000 7f6ade013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:23.311 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:23.310+0000 7f6ade013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:23.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:18:23.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:18:23.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:18:23.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:18:23.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:18:23.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:23.689 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:18:23.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:18:23.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:23.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:24.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:24.647 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:24.645+0000 7f6ade013640 -1 Falling back to public interface 2024-09-17T18:18:25.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:25.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:25.102 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:18:25.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:18:25.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:25.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:25.535 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:26.536 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:26.537 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:18:26.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:26.537 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:18:26.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:26.538 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:26.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:27.455 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:27.453+0000 7f6ade013640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:18:27.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:27.935 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:27.936 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:18:27.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:18:27.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:27.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:28.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:29.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:29.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:29.465 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:18:29.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:18:29.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:29.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:29.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:30.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:30.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:30.891 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:18:30.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:18:30.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:30.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:31.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:32.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:18:32.290 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:18:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:32.290 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:32.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:33.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:33.682 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:18:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:18:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:33.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:34.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:35.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:35.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:35.146 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:18:35.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:18:35.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:35.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:35.539 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:36.540 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:36.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:36.541 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:18:36.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:18:36.541 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:36.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:36.936 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:37.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:37.938 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:37.938 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:18:37.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:18:37.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:37.939 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:38.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:39.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:39.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:39.326 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:18:39.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:18:39.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:39.326 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:40.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:41.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:41.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:41.109 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:18:41.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:18:41.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:41.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:42.075 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:43.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:43.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:43.077 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:18:43.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:18:43.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:43.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:43.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:44.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:44.929 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:44.929 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:18:44.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:18:44.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:44.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:45.317 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:46.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:46.318 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:46.319 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:18:46.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:18:46.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:46.319 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:47.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:48.053 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:18:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:18:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:48.743 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:49.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:49.744 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:49.745 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:18:49.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:18:49.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:49.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:18:50.185 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2426443078,v1:127.0.0.1:6803/2426443078] [v2:127.0.0.1:6804/2426443078,v1:127.0.0.1:6805/2426443078] exists,up 5848a82a-73b9-45ee-b4f9-de2d36a8a99e 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:18:50.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:50.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:18:50.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85a47563-702f-4300-a299-288e029a1d29 2024-09-17T18:18:50.192 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 85a47563-702f-4300-a299-288e029a1d29 2024-09-17T18:18:50.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 85a47563-702f-4300-a299-288e029a1d29' 2024-09-17T18:18:50.192 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:18:50.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKyOlm0yEdDBAACBuAbAefVuiS3jCEvXkuog== 2024-09-17T18:18:50.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKyOlm0yEdDBAACBuAbAefVuiS3jCEvXkuog=="}' 2024-09-17T18:18:50.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85a47563-702f-4300-a299-288e029a1d29 -i td/ec-error-rollforward/1/new.json 2024-09-17T18:18:50.693 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:18:50.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-17T18:18:50.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKyOlm0yEdDBAACBuAbAefVuiS3jCEvXkuog== --osd-uuid 85a47563-702f-4300-a299-288e029a1d29 2024-09-17T18:18:50.732 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:50.729+0000 7f97ddc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:50.742 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:50.740+0000 7f97ddc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:50.743 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:50.741+0000 7f97ddc2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:50.744 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:50.741+0000 7f97ddc2c640 -1 bdev(0x55d292da5c00 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:18:50.744 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:50.741+0000 7f97ddc2c640 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-17T18:18:56.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-17T18:18:56.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:18:56.742 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:18:56.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:18:56.742 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:18:57.277 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:18:57.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:18:57.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:18:57.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:18:57.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:18:57.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:18:57.302 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:57.299+0000 7f40e9d34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:57.320 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:57.318+0000 7f40e9d34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:57.321 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:57.319+0000 7f40e9d34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:18:57.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:18:57.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:18:57.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:18:57.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:57.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:18:58.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:18:58.661 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:18:58.660+0000 7f40e9d34640 -1 Falling back to public interface 2024-09-17T18:18:59.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:18:59.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:18:59.153 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:18:59.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:18:59.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:18:59.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:18:59.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:00.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:00.571 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:19:00.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:00.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:19:00.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:00.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:00.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:01.557 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:01.556+0000 7f40e9d34640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:19:01.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:01.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:01.964 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:19:01.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:19:01.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:01.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:02.385 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:03.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:03.386 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:03.387 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:19:03.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:19:03.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:03.387 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:03.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:04.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:04.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:04.925 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:19:04.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:19:04.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:04.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:05.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:06.414 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:06.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:06.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:19:06.415 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:19:06.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:06.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:07.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:07.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:07.837 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:19:07.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:19:07.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:07.838 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:08.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:09.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:09.238 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:09.238 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:19:09.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:19:09.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:09.239 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:09.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:10.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:10.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:10.674 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:19:10.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:19:10.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:10.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:11.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:12.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:12.215 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:12.215 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:19:12.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:19:12.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:12.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:12.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:13.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:13.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:13.751 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:19:13.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:19:13.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:13.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:14.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:15.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:15.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:15.191 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:19:15.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:19:15.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:15.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:16.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:17.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:17.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:17.355 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:19:17.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:19:17.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:17.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:17.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:18.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:18.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:18.820 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:19:18.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:19:18.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:18.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:19.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:20.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:20.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:20.247 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:19:20.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:19:20.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:20.248 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:20.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:21.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:21.950 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:19:21.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:21.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:19:21.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:21.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:22.946 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:23.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:23.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:23.948 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:19:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:19:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:23.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:24.492 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:25.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:25.493 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:25.494 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:19:25.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:19:25.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:25.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:19:25.943 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3516941032,v1:127.0.0.1:6811/3516941032] [v2:127.0.0.1:6812/3516941032,v1:127.0.0.1:6813/3516941032] exists,up 85a47563-702f-4300-a299-288e029a1d29 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:19:25.947 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:19:25.948 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:19:25.949 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7b70764b-3eb7-4e8d-aee2-1568a64eec07 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7b70764b-3eb7-4e8d-aee2-1568a64eec07' 2024-09-17T18:19:25.950 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 7b70764b-3eb7-4e8d-aee2-1568a64eec07 2024-09-17T18:19:25.951 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:19:25.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAtyOlm9utfORAArWgAuDTxtg+Y6x7y6gBcVg== 2024-09-17T18:19:25.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAtyOlm9utfORAArWgAuDTxtg+Y6x7y6gBcVg=="}' 2024-09-17T18:19:25.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7b70764b-3eb7-4e8d-aee2-1568a64eec07 -i td/ec-error-rollforward/2/new.json 2024-09-17T18:19:26.443 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:19:26.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-17T18:19:26.455 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtyOlm9utfORAArWgAuDTxtg+Y6x7y6gBcVg== --osd-uuid 7b70764b-3eb7-4e8d-aee2-1568a64eec07 2024-09-17T18:19:26.483 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:26.480+0000 7fd4b6bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:26.497 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:26.492+0000 7fd4b6bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:26.497 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:26.493+0000 7fd4b6bcc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:26.497 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:26.493+0000 7fd4b6bcc640 -1 bdev(0x55eecaf97c00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:19:26.497 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:26.493+0000 7fd4b6bcc640 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-17T18:19:32.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-17T18:19:32.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:19:32.671 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:19:32.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:19:32.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:19:33.208 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:19:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:19:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:19:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:19:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:19:33.209 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:19:33.233 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:33.231+0000 7f74b9a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:33.235 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:33.233+0000 7f74b9a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:33.236 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:33.234+0000 7f74b9a9c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:19:33.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:19:33.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:19:33.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:19:33.630 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:33.631 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:34.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:35.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:35.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:35.057 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:19:35.057 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:19:35.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:35.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:35.073 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:35.069+0000 7f74b9a9c640 -1 Falling back to public interface 2024-09-17T18:19:35.474 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:36.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:36.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:36.476 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:19:36.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:19:36.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:36.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:36.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:37.713 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:19:37.710+0000 7f74b9a9c640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:19:37.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:37.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:37.926 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:19:37.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:19:37.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:37.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:38.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:39.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:39.400 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:39.400 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:19:39.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:19:39.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:39.401 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:39.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:40.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:40.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:40.846 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:19:40.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:19:40.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:40.847 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:41.370 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:42.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:42.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:42.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:19:42.372 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:19:42.372 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:42.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:42.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:43.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:43.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:43.835 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:19:43.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:19:43.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:43.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:44.256 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:45.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:45.258 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:19:45.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:45.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:19:45.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:45.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:45.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:46.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:46.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:46.656 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:19:46.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:19:46.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:46.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:47.051 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:48.053 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:48.053 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:19:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:19:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:48.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:48.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:49.576 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:49.577 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:19:49.577 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:19:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:49.578 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:50.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:51.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:51.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:51.141 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:19:51.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:19:51.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:51.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:51.967 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:52.968 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:52.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:52.969 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:19:52.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:19:52.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:52.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:53.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:54.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:54.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:54.655 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:19:54.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:19:54.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:54.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:55.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:56.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:56.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:56.097 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:19:56.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:19:56.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:56.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:56.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:57.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:57.899 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:19:57.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:57.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:19:57.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:57.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:19:58.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:19:59.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:19:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:19:59.714 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:19:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:19:59.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:19:59.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:20:00.456 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:01.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:01.457 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:01.457 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:20:01.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:20:01.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:01.458 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:20:01.895 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2654332030,v1:127.0.0.1:6819/2654332030] [v2:127.0.0.1:6820/2654332030,v1:127.0.0.1:6821/2654332030] exists,up 7b70764b-3eb7-4e8d-aee2-1568a64eec07 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-17T18:20:01.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:20:01.900 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:20:01.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:20:01.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26179672-29d7-449a-bd4e-5820ddaeb160 2024-09-17T18:20:01.903 INFO:tasks.workunit.client.0.smithi157.stdout:add osd3 26179672-29d7-449a-bd4e-5820ddaeb160 2024-09-17T18:20:01.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 26179672-29d7-449a-bd4e-5820ddaeb160' 2024-09-17T18:20:01.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:20:01.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBRyOlmJTeHNhAA3o0hZ7OWykaU+ntQdx8T2A== 2024-09-17T18:20:01.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBRyOlmJTeHNhAA3o0hZ7OWykaU+ntQdx8T2A=="}' 2024-09-17T18:20:01.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26179672-29d7-449a-bd4e-5820ddaeb160 -i td/ec-error-rollforward/3/new.json 2024-09-17T18:20:02.392 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:20:02.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-17T18:20:02.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRyOlmJTeHNhAA3o0hZ7OWykaU+ntQdx8T2A== --osd-uuid 26179672-29d7-449a-bd4e-5820ddaeb160 2024-09-17T18:20:02.432 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:02.430+0000 7f5bd1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:02.445 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:02.443+0000 7f5bd1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:02.446 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:02.444+0000 7f5bd1e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:02.446 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:02.445+0000 7f5bd1e0c640 -1 bdev(0x55d7b4250800 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-17T18:20:02.446 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:02.445+0000 7f5bd1e0c640 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-17T18:20:08.590 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-17T18:20:08.591 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:20:08.591 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd3 key to auth repository 2024-09-17T18:20:08.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T18:20:08.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:20:09.153 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:20:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T18:20:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:20:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:20:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:20:09.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:20:09.178 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:09.176+0000 7fcc0a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:09.180 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:09.178+0000 7fcc0a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:09.181 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:09.179+0000 7fcc0a218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:09.572 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:20:09.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:20:09.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:09.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:09.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:10.777 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:10.775+0000 7fcc0a218640 -1 Falling back to public interface 2024-09-17T18:20:10.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:10.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:10.982 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:20:10.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:20:10.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:10.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:11.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:12.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:12.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:12.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:20:12.417 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:20:12.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:12.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:12.857 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:13.858 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:13.859 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:20:13.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:13.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:20:13.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:13.859 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:13.929 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:13.927+0000 7fcc0a218640 -1 osd.3 0 log_to_monitors true 2024-09-17T18:20:14.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:15.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:15.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:15.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:20:15.255 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:20:15.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:15.255 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:15.812 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:16.813 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:16.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:16.814 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:20:16.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:20:16.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:16.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:17.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:18.362 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:18.362 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:20:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:20:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:18.363 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:18.801 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:19.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:19.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:19.803 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:20:19.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:20:19.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:19.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:20.219 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:21.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:21.220 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:21.221 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:20:21.221 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:20:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:21.222 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:21.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:22.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:22.791 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:22.791 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:20:22.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:20:22.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:22.792 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:23.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:24.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:24.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:24.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:20:24.196 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:20:24.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:24.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:24.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:25.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:25.594 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:20:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:20:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:25.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:26.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:27.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:27.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:27.115 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:20:27.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:20:27.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:27.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:28.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:29.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:29.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:29.423 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:20:29.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:20:29.423 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:29.424 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:30.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:31.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:31.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:31.159 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:20:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:20:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:31.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:32.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:33.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:33.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:33.004 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:20:33.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:20:33.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:33.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:33.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:34.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:34.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:34.748 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:20:34.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:20:34.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:34.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:35.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:36.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:36.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:36.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:20:36.531 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:20:36.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:36.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:37.172 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:20:38.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:20:38.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:38.174 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:20:38.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:20:38.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:38.174 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:38.599 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2521667064,v1:127.0.0.1:6827/2521667064] [v2:127.0.0.1:6828/2521667064,v1:127.0.0.1:6829/2521667064] exists,up 26179672-29d7-449a-bd4e-5820ddaeb160 2024-09-17T18:20:38.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:38.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:38.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:38.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-17T18:20:39.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-17T18:20:39.917 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'ec' already exists 2024-09-17T18:20:39.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-17T18:20:41.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-17T18:20:41.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 95253 2024-09-17T18:20:41.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=98034 2024-09-17T18:20:41.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-17T18:20:41.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 98055' 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 98056' 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 98057' 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-17T18:20:42.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-17T18:20:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-17T18:20:43.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:20:43.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:20:43.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:20:43.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:20:43.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:20:43.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:20:43.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 98034 98055 98056 98057 2024-09-17T18:20:43.729 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:43.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:43.736 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:20:43.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-17T18:20:43.738 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:20:43.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:20:43.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:20:43.739 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-17T18:20:43.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:20:43.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:20:43.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:20:43.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:20:43.763 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:43.760+0000 7fe5b60f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:43.765 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:43.763+0000 7fe5b60f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:43.766 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:43.764+0000 7fe5b60f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:20:44.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:44.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:20:44.162 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:20:44.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:44.162 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:20:44.545 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2426443078,v1:127.0.0.1:6803/2426443078] [v2:127.0.0.1:6804/2426443078,v1:127.0.0.1:6805/2426443078] exists,up 5848a82a-73b9-45ee-b4f9-de2d36a8a99e 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:20:44.549 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:44.550 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:44.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-17T18:20:44.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:20:44.553 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:20:44.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:20:44.553 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-17T18:20:44.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:20:44.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:20:44.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:20:44.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:20:44.572 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:44.569+0000 7f7a1000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:44.583 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:44.581+0000 7f7a1000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:44.584 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:44.582+0000 7f7a1000f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:44.972 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:20:44.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:20:44.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:44.973 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:20:45.357 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:45.355+0000 7fe5b60f1640 -1 Falling back to public interface 2024-09-17T18:20:45.389 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3516941032,v1:127.0.0.1:6811/3516941032] [v2:127.0.0.1:6812/3516941032,v1:127.0.0.1:6813/3516941032] exists,up 85a47563-702f-4300-a299-288e029a1d29 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-17T18:20:45.393 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:20:45.394 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:45.395 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-17T18:20:45.396 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:20:45.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:20:45.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:20:45.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-17T18:20:45.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:20:45.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:20:45.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:20:45.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:20:45.416 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:45.414+0000 7f66e181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:45.418 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:45.416+0000 7f66e181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:45.419 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:45.417+0000 7f66e181a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:45.690 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:45.687+0000 7f7a1000f640 -1 Falling back to public interface 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:45.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:20:45.800 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:20:45.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:45.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:20:46.225 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2654332030,v1:127.0.0.1:6819/2654332030] [v2:127.0.0.1:6820/2654332030,v1:127.0.0.1:6821/2654332030] exists,up 7b70764b-3eb7-4e8d-aee2-1568a64eec07 2024-09-17T18:20:46.228 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:46.229 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' 2024-09-17T18:20:46.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:20:46.231 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:20:46.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=fc48ea78-7d5e-4081-84c1-2e60f8b50049 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90642/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:20:46.232 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-17T18:20:46.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:20:46.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:20:46.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:20:46.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:20:46.252 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:46.249+0000 7ff4115a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:46.254 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:46.251+0000 7ff4115a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:46.255 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:46.253+0000 7ff4115a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:46.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:20:46.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:20:46.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:20:46.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:20:46.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:20:47.041 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2521667064,v1:127.0.0.1:6827/2521667064] [v2:127.0.0.1:6828/2521667064,v1:127.0.0.1:6829/2521667064] exists,up 26179672-29d7-449a-bd4e-5820ddaeb160 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:20:47.043 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:20:47.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:20:47.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:20:47.044 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:20:47.044 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:47.042+0000 7f66e181a640 -1 Falling back to public interface 2024-09-17T18:20:47.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:20:47.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:20:47.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:20:47.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:20:47.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:20:47.127 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:20:47.553 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:20:47.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:20:48.147 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:48.145+0000 7ff4115a2640 -1 Falling back to public interface 2024-09-17T18:20:48.689 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:48.686+0000 7fe5b60f1640 -1 osd.0 26 log_to_monitors true 2024-09-17T18:20:49.845 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:49.843+0000 7f7a1000f640 -1 osd.1 26 log_to_monitors true 2024-09-17T18:20:51.212 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:51.211+0000 7f66e181a640 -1 osd.2 25 log_to_monitors true 2024-09-17T18:20:52.091 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:20:52.089+0000 7ff4115a2640 -1 osd.3 26 log_to_monitors true 2024-09-17T18:20:52.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051587 2024-09-17T18:20:52.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051587 2024-09-17T18:20:52.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587' 2024-09-17T18:20:52.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:20:52.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:20:52.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-17T18:20:52.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-17T18:20:52.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-09-17T18:20:52.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:20:52.803 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:20:52.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-17T18:20:52.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-17T18:20:52.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659' 2024-09-17T18:20:52.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:20:52.999 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:20:53.133 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=3 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 3 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-154618822659 3-3' 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051587 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:20:53.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:20:53.135 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051587 2024-09-17T18:20:53.135 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:20:53.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051587 2024-09-17T18:20:53.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-17T18:20:53.137 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 124554051587 2024-09-17T18:20:53.137 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:20:53.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051587 -lt 124554051587 2024-09-17T18:20:53.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:20:53.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953475 2024-09-17T18:20:53.679 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:20:53.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:20:53.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953475 2024-09-17T18:20:53.681 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:20:53.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-17T18:20:53.682 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 137438953475 2024-09-17T18:20:53.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-17T18:20:53.683 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:20:54.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953475 2024-09-17T18:20:54.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:20:55.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:20:55.165 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:20:55.602 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953475 -lt 137438953475 2024-09-17T18:20:55.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:20:55.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822659 2024-09-17T18:20:55.603 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:20:55.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:20:55.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822659 2024-09-17T18:20:55.605 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:20:55.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-17T18:20:55.606 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 154618822659 2024-09-17T18:20:55.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2024-09-17T18:20:55.606 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:20:56.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-17T18:20:56.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:20:56.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-3 2024-09-17T18:20:56.030 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:20:56.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:20:56.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-3 2024-09-17T18:20:56.031 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:20:56.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=3 2024-09-17T18:20:56.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 3' 2024-09-17T18:20:56.032 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 3 2024-09-17T18:20:56.033 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:20:56.438 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 3 -lt 3 2024-09-17T18:20:56.439 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:20:56.439 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:20:56.439 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:20:56.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:20:56.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:20:56.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:20:57.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:20:57.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:20:57.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:20:57.354 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:20:57.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:20:57.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:20:57.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:20:57.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:20:57.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:20:57.982 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:20:57.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:20:57.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:20:57.983 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:20:57.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:57.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:20:57.984 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:20:57.984 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:57.985 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:20:57.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:20:57.986 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-17T18:20:58.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:20:58.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:58.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90642 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ec-error-rollforward 0 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-17T18:20:58.006 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:20:58.007 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:20:58.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:20:58.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:20:58.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:20:58.008 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:20:58.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:20:58.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:20:58.010 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:20:58.010 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:58.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:20:58.011 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:20:58.011 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:58.012 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:20:58.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:20:58.013 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-17T18:20:58.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:20:58.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.014 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.90642 2024-09-17T18:20:58.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.90642 2024-09-17T18:20:58.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:20:58.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:20:58.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:20:58.017 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:20:58.017 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:20:58.082 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-17T18:20:58.082 DEBUG:teuthology.orchestra.run.smithi157:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-17T18:20:58.144 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:20:58.149 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:20:58.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:20:58.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:20:58.149 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bench 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=6164dfae-9a7f-4e7e-b5c6-bdd3867b7d68 --auth-supported=none ' 2024-09-17T18:20:58.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-17T18:20:58.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-17T18:20:58.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-17T18:20:58.152 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:20:58.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-17T18:20:58.152 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bench 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bench 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:20:58.153 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:20:58.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:20:58.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:20:58.154 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:20:58.154 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:20:58.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:20:58.155 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:20:58.155 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:20:58.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:58.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:20:58.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:20:58.156 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:58.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:20:58.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:20:58.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-17T18:20:58.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:20:58.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.159 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:58.159 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100624 2024-09-17T18:20:58.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:20:58.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:20:58.160 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bench 2024-09-17T18:20:58.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:20:58.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.161 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:58.161 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.100624 2024-09-17T18:20:58.163 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:20:58.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:20:58.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:20:58.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:20:58.163 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bench 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bench/a 2024-09-17T18:20:58.164 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-17T18:20:58.619 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100624/$cluster-$name.asok' 2024-09-17T18:20:58.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100624/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:20:58.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:20:58.797 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.100624/ceph-mon.a.asok 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:20:58.798 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.100624/ceph-mon.a.asok config get fsid 2024-09-17T18:20:58.876 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.100624/ceph-mon.a.asok 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:20:58.877 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.100624/ceph-mon.a.asok config get mon_host 2024-09-17T18:20:58.955 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bench 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bench/x 2024-09-17T18:20:58.956 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:59.801 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:59.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100624/$cluster-$name.asok' 2024-09-17T18:20:59.802 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:20:59.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100624/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bench 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:20:59.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bench/0 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=6164dfae-9a7f-4e7e-b5c6-bdd3867b7d68 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:20:59.831 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.100624/$cluster-$name.asok' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.100624/$cluster-$name.asok' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:20:59.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bench/0 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2 2024-09-17T18:20:59.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2' 2024-09-17T18:20:59.834 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2 2024-09-17T18:20:59.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:20:59.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCLyOlmnKN1MhAAUvss/SJ4uvDscWpOPrnu9A== 2024-09-17T18:20:59.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCLyOlmnKN1MhAAUvss/SJ4uvDscWpOPrnu9A=="}' 2024-09-17T18:20:59.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2 -i td/osd-bench/0/new.json 2024-09-17T18:21:00.120 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:21:00.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bench/0/new.json 2024-09-17T18:21:00.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=6164dfae-9a7f-4e7e-b5c6-bdd3867b7d68 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLyOlmnKN1MhAAUvss/SJ4uvDscWpOPrnu9A== --osd-uuid c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2 2024-09-17T18:21:00.159 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:00.158+0000 7fe4efe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:00.189 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:00.187+0000 7fe4efe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:00.190 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:00.189+0000 7fe4efe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:00.191 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:00.189+0000 7fe4efe0d640 -1 bdev(0x563997601c00 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:21:00.191 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:00.189+0000 7fe4efe0d640 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-17T18:21:06.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-17T18:21:06.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:21:06.127 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:21:06.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:21:06.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:21:06.671 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:21:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:21:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=6164dfae-9a7f-4e7e-b5c6-bdd3867b7d68 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:21:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:21:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:21:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:21:06.696 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:06.695+0000 7fa566789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:06.711 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:06.709+0000 7fa566789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:06.712 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:06.710+0000 7fa566789640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:21:07.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:07.126 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:21:07.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:21:07.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:07.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:07.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:08.060 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:08.059+0000 7fa566789640 -1 Falling back to public interface 2024-09-17T18:21:08.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:08.544 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:21:08.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:08.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:21:08.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:08.544 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:08.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:09.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:09.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:09.962 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:21:09.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:21:09.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:09.962 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:10.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:10.705 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:21:10.704+0000 7fa566789640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:21:11.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:11.353 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:11.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:21:11.354 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:21:11.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:11.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:11.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:12.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:12.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:12.778 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:21:12.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:21:12.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:12.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:13.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:14.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:14.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:14.216 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:21:14.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:21:14.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:14.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:14.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:15.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:15.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:15.612 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:21:15.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:21:15.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:15.612 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:16.027 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:17.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:17.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:17.029 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:21:17.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:21:17.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:17.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:17.561 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:18.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:18.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:18.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:21:18.563 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:21:18.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:18.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:18.989 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:19.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:19.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:19.992 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:21:19.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:21:19.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:19.993 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:20.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:21.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:21.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:21.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:21:21.558 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:21:21.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:21.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:22.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:23.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:23.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:23.020 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:21:23.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:21:23.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:23.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:23.775 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:24.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:24.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:24.777 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:21:24.777 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:21:24.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:24.778 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:26.546 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:26.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:26.547 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:21:26.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:21:26.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:26.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:27.518 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:28.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:28.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:28.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:21:28.520 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:21:28.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:28.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:29.411 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:30.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:30.412 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:30.412 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:21:30.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:21:30.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:30.413 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:31.187 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:32.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:32.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:32.189 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:21:32.189 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:21:32.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:32.190 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:33.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:21:34.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:21:34.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:21:34.062 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:21:34.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:21:34.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:21:34.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:21:34.483 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3885200737,v1:127.0.0.1:6803/3885200737] [v2:127.0.0.1:6804/3885200737,v1:127.0.0.1:6805/3885200737] exists,up c1b6994a-f9c0-4ff7-84dc-78f8f473fdc2 2024-09-17T18:21:34.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:21:34.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:21:34.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:21:34.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-17T18:21:34.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-17T18:21:34.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-17T18:21:34.498 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-17T18:21:34.498 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-17T18:21:34.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-17T18:21:34.512 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-17T18:21:34.513 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-17T18:21:34.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-17T18:21:34.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-17T18:21:34.527 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_max_block_size 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-17T18:21:34.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-17T18:21:34.543 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-17T18:21:34.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-17T18:21:34.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-17T18:21:34.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=bench_small_size_max_iops 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-17T18:21:34.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-17T18:21:34.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-17T18:21:34.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-17T18:21:34.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-17T18:21:34.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-17T18:21:34.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-17T18:21:34.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-17T18:21:34.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-17T18:21:34.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-17T18:21:34.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-17T18:21:34.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-17T18:21:34.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-17T18:21:34.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-17T18:21:34.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-17T18:22:23.983 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_written": 1073741824, 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout: "blocksize": 4194304, 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout: "elapsed_sec": 23.908501515000001, 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_per_sec": 44910460.964119524, 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout: "iops": 10.707488289861566 2024-09-17T18:22:23.984 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:22:23.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-17T18:22:24.358 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_written": 10, 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout: "blocksize": 14456, 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout: "elapsed_sec": 0.033515639999999999, 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_per_sec": 298.36816483289596, 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout: "iops": 0.020639745768739344 2024-09-17T18:22:24.359 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:22:24.368 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_written": 1, 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout: "blocksize": 14456, 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout: "elapsed_sec": 0.034550775999999998, 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout: "bytes_per_sec": 28.942909994264674, 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout: "iops": 0.0020021382121101741 2024-09-17T18:22:24.601 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-17T18:22:24.610 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:22:24.611 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:22:24.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:22:24.716 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:22:24.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:22:24.717 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:22:24.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:22:24.718 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:22:24.718 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:22:24.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.719 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:22:24.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:22:24.720 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.720 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:22:24.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:22:24.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-17T18:22:24.747 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:22:24.747 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:24.747 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:22:24.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100624 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bench 0 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:22:24.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:22:24.750 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:22:24.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:22:24.751 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:22:24.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:22:24.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:22:24.753 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:22:24.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.753 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:22:24.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:22:24.754 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.754 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:22:24.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:22:24.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-17T18:22:24.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:22:24.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:24.757 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.100624 2024-09-17T18:22:24.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.100624 2024-09-17T18:22:24.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:22:24.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:22:24.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:22:24.759 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:22:24.759 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:22:24.823 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-17T18:22:24.823 DEBUG:teuthology.orchestra.run.smithi157:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-17T18:22:24.885 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bluefs-volume-ops 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:24.892 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-17T18:22:24.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-17T18:22:24.893 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_bluestore2 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:TEST_bluestore_expand' 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-17T18:22:24.894 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:22:24.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:22:24.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:22:24.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:22:24.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:22:24.896 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:22:24.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:22:24.897 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:22:24.897 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:22:24.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:22:24.898 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:22:24.899 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:22:24.899 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:22:24.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:22:24.900 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:22:24.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:22:24.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:24.901 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:24.901 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:22:24.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:22:24.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:22:24.902 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-17T18:22:24.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:22:24.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:24.903 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:24.903 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102844 2024-09-17T18:22:24.905 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-17T18:22:24.906 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-17T18:22:24.908 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-17T18:22:24.909 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:25.359 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:22:25.360 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:22:25.552 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:22:25.554 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:22:25.555 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get fsid 2024-09-17T18:22:25.637 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:22:25.638 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get mon_host 2024-09-17T18:22:25.716 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-17T18:22:25.717 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:22:26.565 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:22:26.566 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:22:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:22:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:22:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:22:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:22:26.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:22:26.596 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:22:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:22:26.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:22:26.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:22:26.598 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:22:26.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:22:26.598 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:22:26.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9900589b-54ca-44ac-82b2-67b73c78870b' 2024-09-17T18:22:26.599 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:22:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDiyOlm70h1JBAAo1gXctz4GlwXezxME5o7Jw== 2024-09-17T18:22:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDiyOlm70h1JBAAo1gXctz4GlwXezxME5o7Jw=="}' 2024-09-17T18:22:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9900589b-54ca-44ac-82b2-67b73c78870b -i td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:22:26.960 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:22:26.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:22:26.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiyOlm70h1JBAAo1gXctz4GlwXezxME5o7Jw== --osd-uuid 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:22:26.998 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:26.995+0000 7f5435cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:27.020 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:27.018+0000 7f5435cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:27.021 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:27.019+0000 7f5435cb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:27.022 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:27.019+0000 7f5435cb6640 -1 bdev(0x5602c6ce7c00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:22:27.022 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:27.019+0000 7f5435cb6640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-17T18:22:35.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-17T18:22:35.906 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:22:35.907 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:22:35.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:22:35.907 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:22:36.440 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:22:36.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:22:36.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:22:36.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:22:36.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:22:36.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:22:36.466 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:36.464+0000 7f7a4d9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:36.481 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:36.478+0000 7f7a4d9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:36.482 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:36.479+0000 7f7a4d9ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:36.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:22:36.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:22:36.886 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:36.887 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:37.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:38.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:38.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:38.297 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:22:38.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:22:38.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:38.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:38.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:38.830 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:38.828+0000 7f7a4d9ae640 -1 Falling back to public interface 2024-09-17T18:22:39.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:39.692 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:39.692 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:22:39.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:22:39.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:39.693 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:40.122 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:41.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:41.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:41.124 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:22:41.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:22:41.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:41.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:41.599 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:42.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:42.600 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:42.601 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:22:42.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:22:42.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:42.601 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:43.033 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:43.566 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:22:43.564+0000 7f7a4d9ae640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:22:44.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:44.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:44.035 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:22:44.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:22:44.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:44.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:44.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:45.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:45.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:45.522 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:22:45.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:22:45.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:45.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:45.941 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:46.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:46.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:46.942 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:22:46.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:22:46.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:46.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:47.475 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:48.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:48.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:48.477 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:22:48.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:22:48.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:48.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:48.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:49.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:49.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:49.890 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:22:49.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:22:49.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:49.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:50.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:51.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:51.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:51.339 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:22:51.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:22:51.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:51.339 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:51.732 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:52.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:52.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:52.734 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:22:52.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:22:52.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:52.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:53.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:54.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:54.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:54.227 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:22:54.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:22:54.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:54.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:54.652 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:55.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:55.653 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:55.654 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:22:55.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:22:55.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:55.654 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:56.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:57.115 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:57.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:57.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:22:57.116 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:22:57.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:57.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:22:58.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:22:59.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:22:59.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:22:59.138 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:22:59.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:22:59.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:22:59.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:00.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:01.035 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:01.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:01.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:23:01.036 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:23:01.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:01.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:01.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:02.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:02.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:02.965 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:23:02.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:23:02.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:02.966 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:03.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:04.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:04.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:04.834 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:23:04.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:23:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:04.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:05.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:06.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:06.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:06.522 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-17T18:23:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:23:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:06.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:07.054 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3462636611,v1:127.0.0.1:6803/3462636611] [v2:127.0.0.1:6804/3462636611,v1:127.0.0.1:6805/3462636611] exists,up 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:23:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:23:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:23:07.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:23:07.054 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=104313 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:23:07.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:07.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:07.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 98a75569-9462-4683-ac9e-88daa1cecab8' 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stdout:add osd1 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:23:07.061 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:23:07.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALyelmW9RnBBAAl17VV11qAMmJGrJbBi2GFA== 2024-09-17T18:23:07.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALyelmW9RnBBAAl17VV11qAMmJGrJbBi2GFA=="}' 2024-09-17T18:23:07.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 98a75569-9462-4683-ac9e-88daa1cecab8 -i td/osd-bluefs-volume-ops/1/new.json 2024-09-17T18:23:07.610 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:23:07.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-17T18:23:07.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALyelmW9RnBBAAl17VV11qAMmJGrJbBi2GFA== --osd-uuid 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:23:07.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:07.648+0000 7ff308e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:07.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:07.650+0000 7ff308e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:07.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:07.651+0000 7ff308e1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:07.654 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:07.652+0000 7ff308e1d640 -1 bdev(0x55d82a1bbc00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:23:07.654 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:07.652+0000 7ff308e1d640 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-17T18:23:17.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-17T18:23:17.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:23:17.740 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd1 key to auth repository 2024-09-17T18:23:17.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:23:17.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:23:18.272 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:23:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:23:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:23:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:23:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:23:18.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:23:18.298 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:18.296+0000 7f63a2c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:18.313 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:18.310+0000 7f63a2c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:18.314 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:18.311+0000 7f63a2c0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:23:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:23:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:23:18.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:23:18.751 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:23:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:23:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:18.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:23:18.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:18.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:19.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:20.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:20.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:20.178 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:23:20.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:23:20.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:20.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:20.413 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:20.411+0000 7f63a2c0b640 -1 Falling back to public interface 2024-09-17T18:23:20.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:21.605 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:21.606 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:23:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:21.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:23:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:22.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:23.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:23.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:23.023 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:23:23.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:23:23.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:23.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:23.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:24.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:24.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:24.483 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:23:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:23:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:24.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:25.459 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:25.456+0000 7f63a2c0b640 -1 osd.1 0 log_to_monitors true 2024-09-17T18:23:25.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:25.960 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:25.960 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:23:25.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:23:25.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:25.961 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:26.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:27.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:27.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:27.557 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:23:27.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:23:27.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:27.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:27.978 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:28.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:28.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:28.980 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:23:28.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:23:28.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:28.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:29.510 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:30.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:30.512 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:30.512 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:23:30.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:23:30.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:30.513 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:31.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:32.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:32.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:32.020 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:23:32.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:23:32.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:32.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:32.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:33.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:33.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:33.532 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:23:33.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:23:33.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:33.533 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:33.930 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:34.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:34.931 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:34.931 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:23:34.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:23:34.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:34.932 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:35.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:36.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:36.486 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:23:36.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:36.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:23:36.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:36.487 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:36.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:37.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:37.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:37.944 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:23:37.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:23:37.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:37.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:39.018 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:40.019 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:40.020 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:23:40.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:40.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:23:40.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:40.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:40.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:41.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:41.622 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:41.622 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:23:41.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:23:41.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:41.623 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:42.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:43.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:43.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:43.301 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:23:43.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:23:43.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:43.302 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:44.335 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:45.336 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:45.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:45.337 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:23:45.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:23:45.337 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:45.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:46.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:47.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:47.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:47.062 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:23:47.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:23:47.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:47.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:47.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:48.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:48.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:48.834 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-17T18:23:48.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:23:48.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:48.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:49.254 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3799326203,v1:127.0.0.1:6811/3799326203] [v2:127.0.0.1:6812/3799326203,v1:127.0.0.1:6813/3799326203] exists,up 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:23:49.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:23:49.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:23:49.254 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:23:49.255 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=106867 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:49.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:23:49.260 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:23:49.261 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-17T18:23:49.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:23:49.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:23:49.262 INFO:tasks.workunit.client.0.smithi157.stdout:add osd2 a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:23:49.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a54e7c67-1c45-44f6-85c2-ab26ae7d9a51' 2024-09-17T18:23:49.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:23:49.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1yelmBgVgEBAAvp+WrFvz7agh4EB8P6rEcw== 2024-09-17T18:23:49.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1yelmBgVgEBAAvp+WrFvz7agh4EB8P6rEcw=="}' 2024-09-17T18:23:49.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-17T18:23:49.728 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:23:49.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-17T18:23:49.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1yelmBgVgEBAAvp+WrFvz7agh4EB8P6rEcw== --osd-uuid a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:23:49.769 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:49.767+0000 7f32e300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:49.778 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:49.776+0000 7f32e300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:49.779 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:49.777+0000 7f32e300e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:49.779 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:49.778+0000 7f32e300e640 -1 bdev(0x557ea1c11c00 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:23:49.779 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:23:49.778+0000 7f32e300e640 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-17T18:23:59.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-17T18:23:59.556 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:23:59.557 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:23:59.558 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd2 key to auth repository 2024-09-17T18:23:59.558 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:00.102 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:24:00.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:24:00.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:24:00.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:00.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:00.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:00.128 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:00.127+0000 7fe277730640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:00.140 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:00.138+0000 7fe277730640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:00.141 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:00.139+0000 7fe277730640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:00.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:00.563 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:24:00.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:00.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:00.963 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:01.739 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:01.737+0000 7fe277730640 -1 Falling back to public interface 2024-09-17T18:24:01.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:01.964 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:01.964 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:24:01.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:01.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:01.965 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:02.397 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:03.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:03.398 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:03.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:03.399 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:24:03.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:03.399 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:03.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:04.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:04.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:04.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:04.818 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:24:04.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:04.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:05.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:06.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:06.244 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:24:06.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:06.244 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:24:06.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:06.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:06.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:06.670 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:06.667+0000 7fe277730640 -1 osd.2 0 log_to_monitors true 2024-09-17T18:24:07.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:07.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:07.629 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:24:07.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:24:07.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:07.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:08.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:09.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:09.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:09.145 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:24:09.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:24:09.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:09.146 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:09.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:10.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:10.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:10.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:24:10.628 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:24:10.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:10.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:11.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:12.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:12.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:12.111 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:24:12.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:24:12.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:12.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:12.606 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:13.607 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:13.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:13.608 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:24:13.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:24:13.608 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:13.609 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:14.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:15.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:15.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:15.002 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:24:15.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:24:15.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:15.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:15.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:16.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:16.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:16.418 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:24:16.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:24:16.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:16.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:17.074 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:18.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:18.076 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:18.076 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:24:18.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:24:18.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:18.077 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:18.519 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:19.520 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:19.521 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:24:19.521 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:24:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:19.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:20.322 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:21.323 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:21.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:21.324 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:24:21.324 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:24:21.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:21.325 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:22.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:23.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:23.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:23.128 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:24:23.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:24:23.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:23.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:23.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:24.759 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:24.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:24.760 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:24:24.760 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:24:24.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:24.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:25.782 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:26.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:26.783 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:26.784 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:24:26.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:24:26.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:26.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:27.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:28.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:28.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:28.690 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:24:28.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:24:28.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:28.691 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:29.187 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1396566987,v1:127.0.0.1:6819/1396566987] [v2:127.0.0.1:6820/1396566987,v1:127.0.0.1:6821/1396566987] exists,up a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:24:29.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:24:29.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:24:29.188 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:24:29.188 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-17T18:24:29.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=109421 2024-09-17T18:24:29.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-17T18:24:29.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:24:29.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:24:29.192 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:24:29.193 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:24:29.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:24:29.195 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 23e5d369-031d-4c43-9a77-1e2a6bf5ec55' 2024-09-17T18:24:29.196 INFO:tasks.workunit.client.0.smithi157.stdout:add osd3 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:24:29.196 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:24:29.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdyelmb0RpDBAAhoHTMGpNC1zkIwQKaYRITw== 2024-09-17T18:24:29.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdyelmb0RpDBAAhoHTMGpNC1zkIwQKaYRITw=="}' 2024-09-17T18:24:29.210 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 -i td/osd-bluefs-volume-ops/3/new.json 2024-09-17T18:24:29.663 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:24:29.675 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-17T18:24:29.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdyelmb0RpDBAAhoHTMGpNC1zkIwQKaYRITw== --osd-uuid 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:24:29.704 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:29.701+0000 7fa34a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:29.723 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:29.721+0000 7fa34a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:29.724 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:29.722+0000 7fa34a145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:29.722+0000 7fa34a145640 -1 bdev(0x55a20ce57c00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-17T18:24:29.725 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:29.722+0000 7fa34a145640 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-17T18:24:39.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-17T18:24:39.761 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:24:39.761 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd3 key to auth repository 2024-09-17T18:24:39.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T18:24:39.762 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:40.294 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:24:40.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T18:24:40.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:24:40.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:40.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:40.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:40.321 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:40.319+0000 7f7797418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:40.323 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:40.321+0000 7f7797418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:40.324 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:40.322+0000 7f7797418640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:40.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:40.740 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:24:40.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:40.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:40.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:41.191 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:42.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:42.193 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:42.193 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:24:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:42.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:42.617 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:42.669 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:42.668+0000 7f7797418640 -1 Falling back to public interface 2024-09-17T18:24:43.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:43.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:43.619 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:24:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:43.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:44.043 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:45.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:45.045 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:45.045 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:24:45.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:45.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:45.046 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:45.464 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:24:46.466 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:24:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:46.466 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:46.889 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:47.716 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:24:47.715+0000 7f7797418640 -1 osd.3 0 log_to_monitors true 2024-09-17T18:24:47.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:47.890 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:47.891 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:24:47.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:24:47.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:47.891 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:48.415 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:49.416 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:49.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:49.417 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:24:49.417 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:24:49.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:49.418 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:49.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:50.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:50.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:50.872 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:24:50.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:24:50.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:50.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:51.352 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:52.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:52.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:52.354 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:24:52.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:24:52.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:52.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:52.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:53.869 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:53.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:53.870 INFO:tasks.workunit.client.0.smithi157.stdout:9 2024-09-17T18:24:53.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:24:53.870 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:53.871 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:54.268 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:55.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:55.269 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:55.270 INFO:tasks.workunit.client.0.smithi157.stdout:10 2024-09-17T18:24:55.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:24:55.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:55.270 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:55.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:56.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:56.682 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:56.682 INFO:tasks.workunit.client.0.smithi157.stdout:11 2024-09-17T18:24:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:24:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:56.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:57.230 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:58.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:58.231 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:58.232 INFO:tasks.workunit.client.0.smithi157.stdout:12 2024-09-17T18:24:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:24:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:58.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:58.694 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:59.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:59.695 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:59.696 INFO:tasks.workunit.client.0.smithi157.stdout:13 2024-09-17T18:24:59.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:24:59.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:59.696 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:00.148 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:01.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:01.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:01.150 INFO:tasks.workunit.client.0.smithi157.stdout:14 2024-09-17T18:25:01.150 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:25:01.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:01.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:02.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:03.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:03.306 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:03.307 INFO:tasks.workunit.client.0.smithi157.stdout:15 2024-09-17T18:25:03.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:25:03.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:03.307 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:04.014 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:05.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:05.015 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:05.015 INFO:tasks.workunit.client.0.smithi157.stdout:16 2024-09-17T18:25:05.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:25:05.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:05.016 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:05.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:06.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:06.836 INFO:tasks.workunit.client.0.smithi157.stdout:17 2024-09-17T18:25:06.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:25:06.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:06.837 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:07.303 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:08.304 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:08.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:08.305 INFO:tasks.workunit.client.0.smithi157.stdout:18 2024-09-17T18:25:08.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:25:08.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:08.305 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:09.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:25:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:25:10.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:25:10.088 INFO:tasks.workunit.client.0.smithi157.stdout:19 2024-09-17T18:25:10.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:25:10.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:25:10.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:25:10.514 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3758462932,v1:127.0.0.1:6827/3758462932] [v2:127.0.0.1:6828/3758462932,v1:127.0.0.1:6829/3758462932] exists,up 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:25:10.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:25:10.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:25:10.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:25:10.515 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-17T18:25:10.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=111950 2024-09-17T18:25:10.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-17T18:25:15.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-17T18:25:15.517 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-17T18:25:16.012 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-17T18:25:16.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:25:17.025 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-17T18:25:17.091 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:25:17.091 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-17T18:25:17.092 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_113315 2024-09-17T18:25:17.092 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:25:17.092 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:25:18.091 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 42 26 0.101552 0.101562 0.701239 0.420874 2024-09-17T18:25:19.092 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 92 76 0.14842 0.195312 0.297755 0.382194 2024-09-17T18:25:20.091 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 148 132 0.171856 0.21875 0.285074 0.351206 2024-09-17T18:25:21.092 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 202 186 0.181619 0.210938 0.303235 0.335257 2024-09-17T18:25:22.092 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 260 244 0.190601 0.226562 0.260066 0.321004 2024-09-17T18:25:23.092 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 314 298 0.193986 0.210938 0.300659 0.313531 2024-09-17T18:25:24.092 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 372 356 0.198635 0.226562 0.288503 0.309021 2024-09-17T18:25:25.092 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 426 410 0.200171 0.210938 0.291818 0.305576 2024-09-17T18:25:26.092 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 484 468 0.203101 0.226562 0.278128 0.301798 2024-09-17T18:25:27.092 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 538 522 0.203882 0.210938 0.294708 0.300203 2024-09-17T18:25:28.093 INFO:tasks.workunit.client.0.smithi157.stdout: 11 16 602 586 0.208072 0.25 0.276786 0.297007 2024-09-17T18:25:29.093 INFO:tasks.workunit.client.0.smithi157.stdout: 12 16 660 644 0.209611 0.226562 0.310654 0.294939 2024-09-17T18:25:30.093 INFO:tasks.workunit.client.0.smithi157.stdout: 13 16 724 708 0.212716 0.25 0.244405 0.292448 2024-09-17T18:25:31.093 INFO:tasks.workunit.client.0.smithi157.stdout: 14 16 778 762 0.212587 0.210938 0.294197 0.291418 2024-09-17T18:25:32.093 INFO:tasks.workunit.client.0.smithi157.stdout: 15 16 836 820 0.213517 0.226562 0.24903 0.290106 2024-09-17T18:25:33.093 INFO:tasks.workunit.client.0.smithi157.stdout: 16 16 890 874 0.213353 0.210938 0.285392 0.289435 2024-09-17T18:25:34.093 INFO:tasks.workunit.client.0.smithi157.stdout: 17 16 948 932 0.214129 0.226562 0.276331 0.289742 2024-09-17T18:25:35.093 INFO:tasks.workunit.client.0.smithi157.stdout: 18 16 1002 986 0.21395 0.210938 0.342413 0.289515 2024-09-17T18:25:36.093 INFO:tasks.workunit.client.0.smithi157.stdout: 19 16 1060 1044 0.214612 0.226562 0.27634 0.289237 2024-09-17T18:25:37.094 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T18:25:37.092885+0000 min lat: 0.173826 max lat: 0.701239 avg lat: 0.291516 2024-09-17T18:25:37.094 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:25:37.094 INFO:tasks.workunit.client.0.smithi157.stdout: 20 16 1104 1088 0.212474 0.171875 0.310706 0.291516 2024-09-17T18:25:38.094 INFO:tasks.workunit.client.0.smithi157.stdout: 21 16 1159 1143 0.212586 0.214844 0.16843 0.292593 2024-09-17T18:25:39.094 INFO:tasks.workunit.client.0.smithi157.stdout: 22 16 1211 1195 0.212155 0.203125 0.301614 0.292895 2024-09-17T18:25:40.094 INFO:tasks.workunit.client.0.smithi157.stdout: 23 16 1266 1250 0.21227 0.214844 0.285492 0.292319 2024-09-17T18:25:41.094 INFO:tasks.workunit.client.0.smithi157.stdout: 24 16 1324 1308 0.212864 0.226562 0.319345 0.292364 2024-09-17T18:25:42.094 INFO:tasks.workunit.client.0.smithi157.stdout: 25 16 1379 1363 0.212943 0.214844 0.29177 0.291632 2024-09-17T18:25:43.094 INFO:tasks.workunit.client.0.smithi157.stdout: 26 16 1436 1420 0.213315 0.222656 0.292777 0.291697 2024-09-17T18:25:44.094 INFO:tasks.workunit.client.0.smithi157.stdout: 27 16 1491 1475 0.213371 0.214844 0.300242 0.291961 2024-09-17T18:25:45.095 INFO:tasks.workunit.client.0.smithi157.stdout: 28 16 1548 1532 0.213702 0.222656 0.296525 0.291607 2024-09-17T18:25:46.095 INFO:tasks.workunit.client.0.smithi157.stdout: 29 16 1596 1580 0.212798 0.1875 0.293879 0.291699 2024-09-17T18:25:47.095 INFO:tasks.workunit.client.0.smithi157.stdout: 30 16 1651 1635 0.212865 0.214844 0.339117 0.291961 2024-09-17T18:25:48.095 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 30.2066 2024-09-17T18:25:48.095 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 1651 2024-09-17T18:25:48.095 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-17T18:25:48.095 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.213504 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0260276 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.25 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.101562 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 54 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 6.66307 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 64 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 26 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.291997 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0462621 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.701239 2024-09-17T18:25:48.096 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.168148 2024-09-17T18:25:48.201 INFO:tasks.workunit.client.0.smithi157.stdout:after bench 2024-09-17T18:25:48.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-17T18:25:48.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 104313 2024-09-17T18:25:48.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-17T18:25:49.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 104313 2024-09-17T18:25:49.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (104313) - No such process 2024-09-17T18:25:49.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-17T18:25:50.319 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:25:50.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106867 2024-09-17T18:25:50.331 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-17T18:25:51.332 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106867 2024-09-17T18:25:51.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (106867) - No such process 2024-09-17T18:25:51.333 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-17T18:25:51.735 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:25:51.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 109421 2024-09-17T18:25:51.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-17T18:25:52.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 109421 2024-09-17T18:25:52.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (109421) - No such process 2024-09-17T18:25:52.749 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-17T18:25:53.167 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:25:53.179 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111950 2024-09-17T18:25:53.180 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-17T18:25:54.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111950 2024-09-17T18:25:54.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (111950) - No such process 2024-09-17T18:25:54.181 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-17T18:25:54.717 INFO:tasks.workunit.client.0.smithi157.stderr:osd.3 is already down. 2024-09-17T18:25:54.730 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:25:56.786 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:25:56.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:25:59.109 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:25:59.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:26:01.435 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:01.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:26:03.757 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:03.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-17T18:26:03.776 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-17T18:26:03.799 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:04.893 INFO:tasks.workunit.client.0.smithi157.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-17T18:26:04.893 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-17T18:26:04.893 INFO:tasks.workunit.client.0.smithi157.stdout:2 : device size 0x100000000 : using 0x80512000(2.0 GiB) 2024-09-17T18:26:04.893 INFO:tasks.workunit.client.0.smithi157.stdout:Expanding DB/WAL... 2024-09-17T18:26:04.893 INFO:tasks.workunit.client.0.smithi157.stdout:0 : expanding to 0x536870912 2024-09-17T18:26:04.900 INFO:tasks.workunit.client.0.smithi157.stdout:0 : size label updated to 536870912 2024-09-17T18:26:04.901 INFO:tasks.workunit.client.0.smithi157.stdout:1 : expanding to 0x1073741824 2024-09-17T18:26:04.909 INFO:tasks.workunit.client.0.smithi157.stdout:1 : size label updated to 1073741824 2024-09-17T18:26:04.909 INFO:tasks.workunit.client.0.smithi157.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-17T18:26:05.042 INFO:tasks.workunit.client.0.smithi157.stdout:2 : size label updated to 4294967296 2024-09-17T18:26:10.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-17T18:26:10.085 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-17T18:26:10.111 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:11.202 INFO:tasks.workunit.client.0.smithi157.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-17T18:26:11.202 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-17T18:26:11.202 INFO:tasks.workunit.client.0.smithi157.stdout:2 : device size 0x101000000 : using 0x81534000(2.0 GiB) 2024-09-17T18:26:11.202 INFO:tasks.workunit.client.0.smithi157.stdout:Expanding DB/WAL... 2024-09-17T18:26:11.202 INFO:tasks.workunit.client.0.smithi157.stdout:0 : expanding to 0x536870912 2024-09-17T18:26:11.205 INFO:tasks.workunit.client.0.smithi157.stdout:0 : size label updated to 536870912 2024-09-17T18:26:11.205 INFO:tasks.workunit.client.0.smithi157.stdout:1 : expanding to 0x1073741824 2024-09-17T18:26:11.213 INFO:tasks.workunit.client.0.smithi157.stdout:1 : size label updated to 1073741824 2024-09-17T18:26:11.213 INFO:tasks.workunit.client.0.smithi157.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-17T18:26:11.388 INFO:tasks.workunit.client.0.smithi157.stdout:2 : size label updated to 4311744512 2024-09-17T18:26:16.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-17T18:26:16.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-17T18:26:16.531 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:17.641 INFO:tasks.workunit.client.0.smithi157.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-17T18:26:17.641 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-17T18:26:17.641 INFO:tasks.workunit.client.0.smithi157.stdout:2 : device size 0x100020000 : using 0x80519000(2.0 GiB) 2024-09-17T18:26:17.642 INFO:tasks.workunit.client.0.smithi157.stdout:Expanding DB/WAL... 2024-09-17T18:26:17.642 INFO:tasks.workunit.client.0.smithi157.stdout:0 : expanding to 0x536870912 2024-09-17T18:26:17.652 INFO:tasks.workunit.client.0.smithi157.stdout:0 : size label updated to 536870912 2024-09-17T18:26:17.652 INFO:tasks.workunit.client.0.smithi157.stdout:1 : expanding to 0x1073741824 2024-09-17T18:26:17.661 INFO:tasks.workunit.client.0.smithi157.stdout:1 : size label updated to 1073741824 2024-09-17T18:26:17.661 INFO:tasks.workunit.client.0.smithi157.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-17T18:26:17.819 INFO:tasks.workunit.client.0.smithi157.stdout:2 : size label updated to 4295098368 2024-09-17T18:26:22.924 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-17T18:26:22.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-17T18:26:22.950 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:24.050 INFO:tasks.workunit.client.0.smithi157.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-17T18:26:24.050 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-17T18:26:24.050 INFO:tasks.workunit.client.0.smithi157.stdout:2 : device size 0xfff00000 : using 0x80427000(2.0 GiB) 2024-09-17T18:26:24.050 INFO:tasks.workunit.client.0.smithi157.stdout:Expanding DB/WAL... 2024-09-17T18:26:24.051 INFO:tasks.workunit.client.0.smithi157.stdout:0 : expanding to 0x536870912 2024-09-17T18:26:24.073 INFO:tasks.workunit.client.0.smithi157.stdout:0 : size label updated to 536870912 2024-09-17T18:26:24.073 INFO:tasks.workunit.client.0.smithi157.stdout:1 : expanding to 0x1073741824 2024-09-17T18:26:24.084 INFO:tasks.workunit.client.0.smithi157.stdout:1 : size label updated to 1073741824 2024-09-17T18:26:24.084 INFO:tasks.workunit.client.0.smithi157.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-17T18:26:24.217 INFO:tasks.workunit.client.0.smithi157.stdout:2 : size label updated to 4293918720 2024-09-17T18:26:29.350 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:26:31.669 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:31.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:26:33.993 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:34.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:26:36.317 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:36.327 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:26:38.639 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:38.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-17T18:26:38.674 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:40.706 INFO:tasks.workunit.client.0.smithi157.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-17T18:26:40.706 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-17T18:26:40.707 INFO:tasks.workunit.client.0.smithi157.stdout:2 : device size 0x100000000 : using 0x512000(5.1 MiB) 2024-09-17T18:26:40.708 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-17T18:26:40.730 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:43.542 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-17T18:26:43.547 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:26:45.358 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:45.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-17T18:26:45.388 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:48.203 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-17T18:26:48.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:26:50.026 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:50.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-17T18:26:50.058 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:53.119 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-17T18:26:53.119 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-17T18:26:53.124 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:26:54.437 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:54.505 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-17T18:26:54.505 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:26:55.274 INFO:tasks.workunit.client.0.smithi157.stderr:Migrate to WAL device isn't supported. 2024-09-17T18:26:55.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-17T18:26:55.276 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:26:57.588 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:26:57.596 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-17T18:26:57.618 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:27:00.179 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-17T18:27:00.184 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:27:01.996 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:27:02.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-17T18:27:02.026 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:27:04.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:27:05.901 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:27:05.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:27:05.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:27:05.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:27:05.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:27:05.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:05.914 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:27:05.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:27:05.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:27:05.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:27:05.916 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:27:05.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:27:05.942 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:05.939+0000 7f833180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:05.944 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:05.942+0000 7f833180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:05.945 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:05.943+0000 7f833180b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:06.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:27:06.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:27:06.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:27:06.308 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:06.309 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:06.755 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:06.787 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:06.785+0000 7f833180b640 -1 Falling back to public interface 2024-09-17T18:27:07.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:07.756 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:07.757 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:27:07.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:27:07.757 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:07.758 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:08.167 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:09.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:09.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:09.169 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:27:09.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:27:09.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:09.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:09.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:10.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:10.619 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:10.620 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:27:10.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:27:10.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:10.620 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:11.030 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:11.127 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:11.125+0000 7f833180b640 -1 osd.0 27 log_to_monitors true 2024-09-17T18:27:12.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:12.031 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:12.031 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:27:12.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:27:12.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:12.032 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:12.442 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 37 up_thru 37 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/1610773629,v1:127.0.0.1:6803/1610773629] [v2:127.0.0.1:6804/1610773629,v1:127.0.0.1:6805/1610773629] exists,up 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:27:12.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:27:12.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:27:12.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:27:12.443 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:27:12.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=125459 2024-09-17T18:27:12.446 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-17T18:27:12.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:27:12.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-17T18:27:12.449 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:27:12.450 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:27:12.451 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:27:12.472 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:12.469+0000 7f9970389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:12.474 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:12.472+0000 7f9970389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:12.475 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:12.473+0000 7f9970389640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:27:12.844 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:27:12.845 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:27:12.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:12.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:27:13.287 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:14.081 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:14.078+0000 7f9970389640 -1 Falling back to public interface 2024-09-17T18:27:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:27:14.289 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:27:14.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:14.289 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:27:14.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:15.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:15.738 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:27:15.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:15.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:27:15.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:15.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:27:16.156 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:17.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:17.157 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:17.157 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:27:17.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:27:17.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:17.158 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:27:17.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:17.820 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:17.816+0000 7f9970389640 -1 osd.1 30 log_to_monitors true 2024-09-17T18:27:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:18.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:18.593 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:27:18.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:27:18.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:18.594 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:27:19.097 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/4049179260,v1:127.0.0.1:6811/4049179260] [v2:127.0.0.1:6812/4049179260,v1:127.0.0.1:6813/4049179260] exists,up 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:27:19.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:27:19.097 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:27:19.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:27:19.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=126432 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:27:19.101 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:19.102 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:27:19.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-17T18:27:19.104 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:27:19.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:27:19.127 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:19.124+0000 7ffa20734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:19.129 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:19.126+0000 7ffa20734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:19.130 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:19.127+0000 7ffa20734640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:27:19.644 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:19.645 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:27:19.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:27:19.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:19.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:27:20.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:20.470 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:20.468+0000 7ffa20734640 -1 Falling back to public interface 2024-09-17T18:27:21.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:21.063 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:21.064 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:27:21.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:27:21.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:21.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:27:21.488 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:22.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:22.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:22.490 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:27:22.490 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:27:22.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:22.491 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:27:22.913 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:23.592 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:23.590+0000 7ffa20734640 -1 osd.2 32 log_to_monitors true 2024-09-17T18:27:23.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:23.915 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:23.915 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:27:23.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:27:23.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:23.916 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:27:24.478 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 44 up_thru 31 down_at 33 last_clean_interval [15,32) [v2:127.0.0.1:6818/1903199706,v1:127.0.0.1:6819/1903199706] [v2:127.0.0.1:6820/1903199706,v1:127.0.0.1:6821/1903199706] exists,up a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:27:24.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:27:24.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:27:24.479 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:27:24.479 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-17T18:27:24.482 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=127405 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:27:24.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:27:24.484 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-17T18:27:24.485 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:27:24.486 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:27:24.509 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:24.507+0000 7f06fc1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:24.511 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:24.509+0000 7f06fc1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:24.512 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:24.510+0000 7f06fc1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:25.271 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:27:25.272 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:27:25.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:25.272 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:27:25.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:25.861 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:25.860+0000 7f06fc1a8640 -1 Falling back to public interface 2024-09-17T18:27:26.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:26.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:26.739 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:27:26.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:27:26.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:26.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:27:27.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:28.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:28.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:28.207 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:27:28.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:27:28.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:28.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:27:28.639 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:29.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:29.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:29.641 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:27:29.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:27:29.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:29.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:27:30.036 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:30.341 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:27:30.339+0000 7f06fc1a8640 -1 osd.3 34 log_to_monitors true 2024-09-17T18:27:31.037 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:27:31.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:31.038 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:27:31.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:27:31.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:31.038 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:27:31.626 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 48 up_thru 33 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/2165619697,v1:127.0.0.1:6827/2165619697] [v2:127.0.0.1:6828/2165619697,v1:127.0.0.1:6829/2165619697] exists,up 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:27:31.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:27:31.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:27:31.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:27:31.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-17T18:27:31.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=128348 2024-09-17T18:27:31.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-17T18:27:31.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:27:31.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:27:31.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:27:31.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:27:31.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:27:32.227 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:27:32.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789959 2024-09-17T18:27:32.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789959 2024-09-17T18:27:32.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789959' 2024-09-17T18:27:32.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:27:32.473 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:27:32.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-17T18:27:32.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-17T18:27:32.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789959 1-171798691845' 2024-09-17T18:27:32.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:27:32.586 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:27:32.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561028 2024-09-17T18:27:32.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561028 2024-09-17T18:27:32.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789959 1-171798691845 2-188978561028' 2024-09-17T18:27:32.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:27:32.721 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:27:32.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430211 2024-09-17T18:27:32.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430211 2024-09-17T18:27:32.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789959 1-171798691845 2-188978561028 3-206158430211' 2024-09-17T18:27:32.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:27:32.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789959 2024-09-17T18:27:32.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:27:32.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:27:32.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789959 2024-09-17T18:27:32.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:27:32.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789959 2024-09-17T18:27:32.836 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789959' 2024-09-17T18:27:32.836 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 158913789959 2024-09-17T18:27:32.836 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:27:33.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789959 -lt 158913789959 2024-09-17T18:27:33.213 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:27:33.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691845 2024-09-17T18:27:33.213 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:27:33.214 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:27:33.215 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691845 2024-09-17T18:27:33.215 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:27:33.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-17T18:27:33.216 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 171798691845 2024-09-17T18:27:33.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-17T18:27:33.216 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:27:33.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-17T18:27:33.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:27:33.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-188978561028 2024-09-17T18:27:33.642 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:27:33.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:27:33.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-188978561028 2024-09-17T18:27:33.644 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:27:33.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561028 2024-09-17T18:27:33.645 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 188978561028 2024-09-17T18:27:33.645 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 188978561028' 2024-09-17T18:27:33.645 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:27:34.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561026 -lt 188978561028 2024-09-17T18:27:34.068 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:27:35.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:27:35.069 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:27:35.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561029 -lt 188978561028 2024-09-17T18:27:35.459 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:27:35.460 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-206158430211 2024-09-17T18:27:35.460 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:27:35.461 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:27:35.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-206158430211 2024-09-17T18:27:35.461 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:27:35.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430211 2024-09-17T18:27:35.462 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 206158430211' 2024-09-17T18:27:35.463 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 206158430211 2024-09-17T18:27:35.463 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:27:35.888 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430211 -lt 206158430211 2024-09-17T18:27:35.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:27:35.888 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:27:35.889 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:27:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T18:27:36.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:27:36.409 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:27:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:27:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:27:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:27:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:27:36.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:27:36.827 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T18:27:36.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:27:36.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:27:36.828 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:27:37.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T18:27:37.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:27:37.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:27:37.351 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-17T18:27:37.398 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:27:37.399 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-17T18:27:37.399 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_129689 2024-09-17T18:27:37.399 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:27:37.399 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:27:38.399 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 57 41 0.160143 0.160156 0.283537 0.261034 2024-09-17T18:27:39.399 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 112 96 0.18748 0.214844 0.187761 0.310568 2024-09-17T18:27:40.399 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 175 159 0.207011 0.246094 0.301959 0.28536 2024-09-17T18:27:41.399 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 241 225 0.219706 0.257812 0.268668 0.279887 2024-09-17T18:27:42.399 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 300 284 0.221855 0.230469 0.295651 0.274191 2024-09-17T18:27:43.399 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 357 341 0.221983 0.222656 0.225157 0.270633 2024-09-17T18:27:44.399 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 413 397 0.221516 0.21875 0.310323 0.275363 2024-09-17T18:27:45.399 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 473 457 0.223119 0.234375 0.198701 0.276392 2024-09-17T18:27:46.400 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 529 513 0.222631 0.21875 0.217969 0.275182 2024-09-17T18:27:47.400 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 589 573 0.223803 0.234375 0.216796 0.276195 2024-09-17T18:27:48.400 INFO:tasks.workunit.client.0.smithi157.stdout: 11 16 645 629 0.223341 0.21875 0.19152 0.275634 2024-09-17T18:27:49.400 INFO:tasks.workunit.client.0.smithi157.stdout: 12 16 701 685 0.222956 0.21875 0.251893 0.276353 2024-09-17T18:27:50.400 INFO:tasks.workunit.client.0.smithi157.stdout: 13 16 760 744 0.223531 0.230469 0.179352 0.276982 2024-09-17T18:27:51.400 INFO:tasks.workunit.client.0.smithi157.stdout: 14 16 814 798 0.22263 0.210938 0.341513 0.27662 2024-09-17T18:27:52.400 INFO:tasks.workunit.client.0.smithi157.stdout: 15 16 875 859 0.223671 0.238281 0.185149 0.275747 2024-09-17T18:27:53.400 INFO:tasks.workunit.client.0.smithi157.stdout: 16 16 932 916 0.223607 0.222656 0.150242 0.276186 2024-09-17T18:27:54.401 INFO:tasks.workunit.client.0.smithi157.stdout: 17 16 998 982 0.225617 0.257812 0.157588 0.275525 2024-09-17T18:27:55.401 INFO:tasks.workunit.client.0.smithi157.stdout: 18 16 1071 1055 0.228923 0.285156 0.139493 0.271692 2024-09-17T18:27:56.401 INFO:tasks.workunit.client.0.smithi157.stdout: 19 16 1145 1129 0.232087 0.289062 0.237915 0.267401 2024-09-17T18:27:57.401 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T18:27:57.400002+0000 min lat: 0.108013 max lat: 0.527371 avg lat: 0.264385 2024-09-17T18:27:57.401 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:27:57.401 INFO:tasks.workunit.client.0.smithi157.stdout: 20 16 1219 1203 0.234934 0.289062 0.209708 0.264385 2024-09-17T18:27:58.401 INFO:tasks.workunit.client.0.smithi157.stdout: 21 16 1299 1283 0.238625 0.3125 0.2618 0.260714 2024-09-17T18:27:59.401 INFO:tasks.workunit.client.0.smithi157.stdout: 22 16 1355 1339 0.237721 0.21875 0.334683 0.25973 2024-09-17T18:28:00.401 INFO:tasks.workunit.client.0.smithi157.stdout: 23 16 1434 1418 0.240801 0.308594 0.189365 0.258508 2024-09-17T18:28:01.401 INFO:tasks.workunit.client.0.smithi157.stdout: 24 16 1503 1487 0.241997 0.269531 0.217648 0.257202 2024-09-17T18:28:02.401 INFO:tasks.workunit.client.0.smithi157.stdout: 25 16 1571 1555 0.24294 0.265625 0.25951 0.255878 2024-09-17T18:28:03.402 INFO:tasks.workunit.client.0.smithi157.stdout: 26 16 1635 1619 0.24321 0.25 0.234664 0.255767 2024-09-17T18:28:04.402 INFO:tasks.workunit.client.0.smithi157.stdout: 27 16 1703 1687 0.24404 0.265625 0.200231 0.254843 2024-09-17T18:28:05.402 INFO:tasks.workunit.client.0.smithi157.stdout: 28 16 1782 1766 0.246344 0.308594 0.167335 0.253057 2024-09-17T18:28:06.402 INFO:tasks.workunit.client.0.smithi157.stdout: 29 16 1862 1846 0.248624 0.3125 0.22029 0.250547 2024-09-17T18:28:07.402 INFO:tasks.workunit.client.0.smithi157.stdout: 30 16 1932 1916 0.249451 0.273438 0.276371 0.249522 2024-09-17T18:28:08.402 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 30.1794 2024-09-17T18:28:08.402 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 1932 2024-09-17T18:28:08.402 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-17T18:28:08.402 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-17T18:28:08.402 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.250067 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0367261 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.3125 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.160156 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 64 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 9.40188 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 80 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 41 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.249527 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0594109 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.527371 2024-09-17T18:28:08.403 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.108013 2024-09-17T18:28:08.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125459 2024-09-17T18:28:08.481 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-17T18:28:09.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125459 2024-09-17T18:28:09.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (125459) - No such process 2024-09-17T18:28:09.483 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-17T18:28:09.939 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:28:09.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126432 2024-09-17T18:28:09.951 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-17T18:28:10.952 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126432 2024-09-17T18:28:10.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (126432) - No such process 2024-09-17T18:28:10.953 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-17T18:28:11.418 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:28:11.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127405 2024-09-17T18:28:11.430 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-17T18:28:12.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127405 2024-09-17T18:28:12.431 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (127405) - No such process 2024-09-17T18:28:12.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-17T18:28:12.840 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:28:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128348 2024-09-17T18:28:12.851 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-17T18:28:13.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128348 2024-09-17T18:28:13.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (128348) - No such process 2024-09-17T18:28:13.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-17T18:28:14.349 INFO:tasks.workunit.client.0.smithi157.stderr:osd.3 is already down. 2024-09-17T18:28:14.361 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:28:16.168 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:16.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-17T18:28:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:512+0 records in 2024-09-17T18:28:16.434 INFO:tasks.workunit.client.0.smithi157.stderr:512+0 records out 2024-09-17T18:28:16.434 INFO:tasks.workunit.client.0.smithi157.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.253972 s, 2.1 GB/s 2024-09-17T18:28:16.434 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-17T18:28:16.453 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:21.505 INFO:tasks.workunit.client.0.smithi157.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-17T18:28:21.509 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:28:23.822 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:23.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:28:25.644 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:25.651 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-17T18:28:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:28:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:28:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.490128 s, 2.2 GB/s 2024-09-17T18:28:26.143 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-17T18:28:26.163 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:34.717 INFO:tasks.workunit.client.0.smithi157.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-17T18:28:34.721 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-17T18:28:34.744 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:35.803 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:35.802+0000 7f330ec16980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:36.567 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:36.565+0000 7f330ec16980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:37.817 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:28:38.372 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:38.371+0000 7f8623510980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:39.148 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:39.146+0000 7f8623510980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:40.140 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:40.151 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:28:41.466 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:41.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-17T18:28:41.492 INFO:tasks.workunit.client.0.smithi157.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-17T18:28:41.493 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:42.787 INFO:tasks.workunit.client.0.smithi157.stdout:DB device added 2024-09-17T18:28:42.790 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-17T18:28:42.814 INFO:tasks.workunit.client.0.smithi157.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-17T18:28:42.814 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:43.606 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:43.603+0000 7f82fcddc980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:44.133 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:44.130+0000 7f82fcddc980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:44.184 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:44.181+0000 7f82fcddc980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:44.865 INFO:tasks.workunit.client.0.smithi157.stdout:WAL device added 2024-09-17T18:28:44.868 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-17T18:28:44.891 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:45.951 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:45.948+0000 7fbefaa86980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:46.727 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:46.725+0000 7fbefaa86980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:47.970 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:28:48.521 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:48.519+0000 7feaeb416980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:49.286 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:49.283+0000 7feaeb416980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:28:50.283 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:50.291 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:28:52.108 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:52.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-17T18:28:52.140 INFO:tasks.workunit.client.0.smithi157.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-17T18:28:52.140 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:54.192 INFO:tasks.workunit.client.0.smithi157.stdout:WAL device added 2024-09-17T18:28:54.196 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-17T18:28:54.221 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:28:57.282 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-17T18:28:57.313 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:28:59.099 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:28:59.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:28:59.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:28:59.111 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:28:59.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:28:59.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:28:59.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-17T18:28:59.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:28:59.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:28:59.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:28:59.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:28:59.136 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:59.134+0000 7f3d987cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:59.139 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:59.136+0000 7f3d987cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:59.139 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:28:59.137+0000 7f3d987cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:59.528 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:28:59.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:28:59.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:59.529 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:59.942 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:00.724 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:00.722+0000 7f3d987cc640 -1 Falling back to public interface 2024-09-17T18:29:00.943 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:00.944 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:29:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:00.944 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:01.371 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:02.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:02.373 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:02.373 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:29:02.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:02.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:02.374 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:02.814 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:03.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:03.815 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:03.815 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:29:03.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:03.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:03.816 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:04.225 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:05.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:05.226 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:05.227 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:29:05.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:29:05.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:05.227 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:05.589 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:05.587+0000 7f3d987cc640 -1 osd.0 50 log_to_monitors true 2024-09-17T18:29:05.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:06.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:06.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:06.671 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:29:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:29:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:06.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:07.083 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 61 up_thru 61 down_at 51 last_clean_interval [37,50) [v2:127.0.0.1:6802/528310271,v1:127.0.0.1:6803/528310271] [v2:127.0.0.1:6804/528310271,v1:127.0.0.1:6805/528310271] exists,up 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:29:07.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:07.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:07.083 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:07.084 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=135839 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-17T18:29:07.087 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:29:07.088 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-17T18:29:07.089 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:29:07.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:29:07.090 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:29:07.090 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-17T18:29:07.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:29:07.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:29:07.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:29:07.091 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:29:07.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:07.110+0000 7fd3c6b32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:07.114 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:07.112+0000 7fd3c6b32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:07.115 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:07.113+0000 7fd3c6b32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:07.502 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:29:07.503 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:29:07.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:07.503 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:07.969 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:08.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:08.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:08.971 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:29:08.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:08.971 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:08.972 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:09.205 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:09.202+0000 7fd3c6b32640 -1 Falling back to public interface 2024-09-17T18:29:09.403 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:10.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:10.405 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:29:10.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:10.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:10.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:10.405 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:10.798 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:11.100 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:11.097+0000 7fd3c6b32640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:11.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:11.799 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:11.800 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:29:11.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:11.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:11.800 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:11.867 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:11.864+0000 7fd3c6b32640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:12.232 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:12.871 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:12.870+0000 7fd3c6b32640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:13.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:13.233 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:13.234 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:29:13.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:29:13.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:13.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:13.616 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:13.654 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:13.653+0000 7fd3c6b32640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:14.041 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:14.040+0000 7fd3c6b32640 -1 osd.1 53 log_to_monitors true 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:14.618 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:15.040 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 64 up_thru 64 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/1339122343,v1:127.0.0.1:6811/1339122343] [v2:127.0.0.1:6812/1339122343,v1:127.0.0.1:6813/1339122343] exists,up 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:29:15.040 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:15.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:15.041 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:15.041 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=136843 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:29:15.058 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:29:15.059 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-17T18:29:15.060 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:29:15.061 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:29:15.061 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:29:15.062 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-17T18:29:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:29:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:29:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:29:15.062 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:29:15.084 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:15.082+0000 7f2d14020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:15.140 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:15.139+0000 7f2d14020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:15.141 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:15.140+0000 7f2d14020640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:15.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:29:15.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:29:15.471 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:29:15.472 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:15.473 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:15.872 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:16.515 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:16.514+0000 7f2d14020640 -1 Falling back to public interface 2024-09-17T18:29:16.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:16.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:16.874 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:29:16.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:16.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:16.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:17.298 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:18.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:18.178+0000 7f2d14020640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:18.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:18.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:18.300 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:29:18.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:18.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:18.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:18.748 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:18.942 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:18.940+0000 7f2d14020640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:19.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:19.750 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:19.751 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:29:19.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:19.751 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:19.752 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:20.176 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:20.205 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:20.203+0000 7f2d14020640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:20.967 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:20.966+0000 7f2d14020640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:29:21.177 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:21.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:21.178 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:29:21.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:29:21.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:21.178 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:21.572 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:21.905 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:21.903+0000 7f2d14020640 -1 osd.2 56 log_to_monitors true 2024-09-17T18:29:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:22.574 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:29:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:29:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:22.575 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:29:23.245 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 68 up_thru 68 down_at 57 last_clean_interval [44,56) [v2:127.0.0.1:6818/3853363639,v1:127.0.0.1:6819/3853363639] [v2:127.0.0.1:6820/3853363639,v1:127.0.0.1:6821/3853363639] exists,up a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:29:23.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:23.245 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:23.246 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:23.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=137863 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:29:23.249 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:29:23.250 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-17T18:29:23.251 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:29:23.252 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:29:23.252 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:29:23.252 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-17T18:29:23.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:29:23.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:29:23.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:29:23.253 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:29:23.275 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:23.273+0000 7f39ac213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:23.277 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:23.275+0000 7f39ac213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:23.278 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:23.276+0000 7f39ac213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:29:23.767 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:23.768 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:29:23.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:29:23.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:23.768 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:29:24.200 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:24.897 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:24.896+0000 7f39ac213640 -1 Falling back to public interface 2024-09-17T18:29:25.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:25.202 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:25.202 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:29:25.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:25.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:25.203 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:29:25.595 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:26.597 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:29:26.597 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:26.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:26.598 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:29:27.028 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:28.029 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:28.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:28.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:28.104 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:29:28.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:28.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:29:28.419 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:29.204 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:29:29.202+0000 7f39ac213640 -1 osd.3 58 log_to_monitors true 2024-09-17T18:29:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:29.421 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:29:29.421 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:29:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:29.422 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:29:29.845 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 72 up_thru 57 down_at 59 last_clean_interval [48,58) [v2:127.0.0.1:6826/2599056600,v1:127.0.0.1:6827/2599056600] [v2:127.0.0.1:6828/2599056600,v1:127.0.0.1:6829/2599056600] exists,up 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:29:29.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:29.845 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:29.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:29.846 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-17T18:29:29.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=138849 2024-09-17T18:29:29.846 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-17T18:29:29.877 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:29:29.877 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-17T18:29:29.877 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_139807 2024-09-17T18:29:29.877 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:29:29.878 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:29:30.877 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 17 1 0.00390599 0.00390625 0.238438 0.238438 2024-09-17T18:29:31.878 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 59 43 0.0839755 0.164062 0.285938 0.692613 2024-09-17T18:29:32.878 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 109 93 0.121082 0.195312 0.35423 0.483758 2024-09-17T18:29:33.878 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 158 142 0.138656 0.191406 0.33161 0.431598 2024-09-17T18:29:34.878 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 210 194 0.151545 0.203125 0.184422 0.39582 2024-09-17T18:29:35.878 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 266 250 0.16274 0.21875 0.29332 0.373919 2024-09-17T18:29:36.878 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 324 308 0.171853 0.226562 0.203499 0.35521 2024-09-17T18:29:37.878 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 376 360 0.175758 0.203125 0.195938 0.346645 2024-09-17T18:29:38.879 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 426 410 0.177929 0.195312 0.325573 0.344041 2024-09-17T18:29:39.879 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 475 459 0.179274 0.191406 0.332654 0.340416 2024-09-17T18:29:40.879 INFO:tasks.workunit.client.0.smithi157.stdout: 11 16 531 515 0.18286 0.21875 0.3517 0.334653 2024-09-17T18:29:41.879 INFO:tasks.workunit.client.0.smithi157.stdout: 12 16 587 571 0.185848 0.21875 0.308113 0.332207 2024-09-17T18:29:42.879 INFO:tasks.workunit.client.0.smithi157.stdout: 13 16 634 618 0.185673 0.183594 0.288639 0.332043 2024-09-17T18:29:43.879 INFO:tasks.workunit.client.0.smithi157.stdout: 14 16 680 664 0.185243 0.179688 0.368545 0.334477 2024-09-17T18:29:44.879 INFO:tasks.workunit.client.0.smithi157.stdout: 15 16 729 713 0.185653 0.191406 0.348711 0.333644 2024-09-17T18:29:45.880 INFO:tasks.workunit.client.0.smithi157.stdout: 16 16 780 764 0.186499 0.199219 0.299078 0.330914 2024-09-17T18:29:46.880 INFO:tasks.workunit.client.0.smithi157.stdout: 17 16 833 817 0.187705 0.207031 0.184095 0.329514 2024-09-17T18:29:47.880 INFO:tasks.workunit.client.0.smithi157.stdout: 18 16 884 868 0.188343 0.199219 0.341137 0.328099 2024-09-17T18:29:48.880 INFO:tasks.workunit.client.0.smithi157.stdout: 19 16 931 915 0.188091 0.183594 0.326418 0.329493 2024-09-17T18:29:49.880 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T18:29:49.879419+0000 min lat: 0.163389 max lat: 1.62389 avg lat: 0.328614 2024-09-17T18:29:49.880 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:29:49.880 INFO:tasks.workunit.client.0.smithi157.stdout: 20 16 981 965 0.188451 0.195312 0.354161 0.328614 2024-09-17T18:29:50.880 INFO:tasks.workunit.client.0.smithi157.stdout: 21 16 1033 1017 0.189148 0.203125 0.349841 0.328275 2024-09-17T18:29:51.881 INFO:tasks.workunit.client.0.smithi157.stdout: 22 16 1079 1063 0.188717 0.179688 0.380117 0.328684 2024-09-17T18:29:52.881 INFO:tasks.workunit.client.0.smithi157.stdout: 23 16 1125 1109 0.188323 0.179688 0.358379 0.328451 2024-09-17T18:29:53.881 INFO:tasks.workunit.client.0.smithi157.stdout: 24 16 1168 1152 0.187475 0.167969 0.352227 0.330788 2024-09-17T18:29:54.881 INFO:tasks.workunit.client.0.smithi157.stdout: 25 16 1225 1209 0.188881 0.222656 0.335616 0.329164 2024-09-17T18:29:55.881 INFO:tasks.workunit.client.0.smithi157.stdout: 26 16 1276 1260 0.189277 0.199219 0.366869 0.328316 2024-09-17T18:29:56.881 INFO:tasks.workunit.client.0.smithi157.stdout: 27 16 1326 1310 0.1895 0.195312 0.358601 0.327766 2024-09-17T18:29:57.881 INFO:tasks.workunit.client.0.smithi157.stdout: 28 16 1373 1357 0.189288 0.183594 0.35041 0.327653 2024-09-17T18:29:58.881 INFO:tasks.workunit.client.0.smithi157.stdout: 29 16 1424 1408 0.189629 0.199219 0.375743 0.328044 2024-09-17T18:29:59.882 INFO:tasks.workunit.client.0.smithi157.stdout: 30 16 1472 1456 0.189557 0.1875 0.342033 0.328288 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 30.2221 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 1472 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.190258 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0382283 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.226562 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.00390625 2024-09-17T18:30:00.882 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 48 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 9.78646 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 58 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 1 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.327505 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.125876 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 1.62389 2024-09-17T18:30:00.883 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.163389 2024-09-17T18:30:01.020 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135839 2024-09-17T18:30:01.021 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-17T18:30:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135839 2024-09-17T18:30:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (135839) - No such process 2024-09-17T18:30:02.022 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-17T18:30:02.560 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:30:02.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136843 2024-09-17T18:30:02.571 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-17T18:30:03.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136843 2024-09-17T18:30:03.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (136843) - No such process 2024-09-17T18:30:03.573 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-17T18:30:04.071 INFO:tasks.workunit.client.0.smithi157.stderr:osd.1 is already down. 2024-09-17T18:30:04.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137863 2024-09-17T18:30:04.082 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-17T18:30:05.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137863 2024-09-17T18:30:05.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (137863) - No such process 2024-09-17T18:30:05.084 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-17T18:30:05.511 INFO:tasks.workunit.client.0.smithi157.stderr:osd.2 is already down. 2024-09-17T18:30:05.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138849 2024-09-17T18:30:05.523 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-17T18:30:06.524 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138849 2024-09-17T18:30:06.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (138849) - No such process 2024-09-17T18:30:06.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-17T18:30:07.123 INFO:tasks.workunit.client.0.smithi157.stderr:osd.3 is already down. 2024-09-17T18:30:07.134 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:30:09.442 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:09.454 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-17T18:30:09.958 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:30:09.958 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:30:09.958 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.502441 s, 2.1 GB/s 2024-09-17T18:30:09.958 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-17T18:30:09.977 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:30:19.798 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-17T18:30:19.798 INFO:tasks.workunit.client.0.smithi157.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-17T18:30:19.802 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:30:22.112 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:22.123 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-17T18:30:22.381 INFO:tasks.workunit.client.0.smithi157.stderr:512+0 records in 2024-09-17T18:30:22.381 INFO:tasks.workunit.client.0.smithi157.stderr:512+0 records out 2024-09-17T18:30:22.381 INFO:tasks.workunit.client.0.smithi157.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.256197 s, 2.1 GB/s 2024-09-17T18:30:22.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-17T18:30:22.399 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:30:28.471 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-17T18:30:28.471 INFO:tasks.workunit.client.0.smithi157.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-17T18:30:28.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-17T18:30:28.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-17T18:30:30.844 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:30.853 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:30:31.417 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:31.414+0000 7f4844a2d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:32.175 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:32.172+0000 7f4844a2d980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:33.164 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:33.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-17T18:30:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:30:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:30:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.495272 s, 2.2 GB/s 2024-09-17T18:30:33.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-17T18:30:33.690 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:30:34.768 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:34.765+0000 7f783ce19980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:35.531 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:35.528+0000 7f783ce19980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:41.927 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:41.924+0000 7f783ce19980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:43.517 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-17T18:30:43.517 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-17T18:30:43.517 INFO:tasks.workunit.client.0.smithi157.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-17T18:30:43.522 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-17T18:30:43.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:30:44.185 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:44.183+0000 7f6066119980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:44.711 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:44.708+0000 7f6066119980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:45.186 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:45.194 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-17T18:30:45.213 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:30:46.010 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:46.008+0000 7f410a7d1980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:46.523 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:46.521+0000 7f410a7d1980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:47.269 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-17T18:30:47.273 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-17T18:30:47.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-17T18:30:48.694 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:48.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-17T18:30:50.503 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:50.511 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-17T18:30:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:30:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:30:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.4948 s, 2.2 GB/s 2024-09-17T18:30:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-17T18:30:51.028 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:30:51.819 INFO:tasks.workunit.client.0.smithi157.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-17T18:30:51.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-17T18:30:51.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:30:52.373 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:52.371+0000 7fbb8fa1c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:53.140 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:30:53.137+0000 7fbb8fa1c980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:30:54.133 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:30:54.140 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-17T18:31:01.246 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:31:01.247 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:31:01.247 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.98478 s, 154 MB/s 2024-09-17T18:31:01.247 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-17T18:31:01.271 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:31:02.333 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:02.331+0000 7fac29f67980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:03.101 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:03.100+0000 7fac29f67980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:03.137 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:03.136+0000 7fac29f67980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:04.342 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-17T18:31:04.342 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-17T18:31:04.342 INFO:tasks.workunit.client.0.smithi157.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-17T18:31:04.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-17T18:31:04.905 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:04.903+0000 7f737eb07980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:05.417 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:05.416+0000 7f737eb07980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:06.164 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:31:06.173 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-17T18:31:06.669 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records in 2024-09-17T18:31:06.669 INFO:tasks.workunit.client.0.smithi157.stderr:1024+0 records out 2024-09-17T18:31:06.669 INFO:tasks.workunit.client.0.smithi157.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.495263 s, 2.2 GB/s 2024-09-17T18:31:06.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-17T18:31:06.689 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:31:15.996 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-17T18:31:15.997 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-17T18:31:15.997 INFO:tasks.workunit.client.0.smithi157.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-17T18:31:16.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-17T18:31:16.562 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:16.560+0000 7f8d2862b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:17.086 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:17.085+0000 7f8d2862b980 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:17.820 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:31:17.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:31:17.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:31:17.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:31:17.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:31:17.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:31:17.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:31:17.834 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:31:17.835 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:31:17.858 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:17.856+0000 7fde94e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:17.860 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:17.859+0000 7fde94e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:17.861 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:17.860+0000 7fde94e1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:18.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:31:18.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:31:18.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:31:18.242 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:18.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:18.659 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:19.452 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:19.450+0000 7fde94e1e640 -1 Falling back to public interface 2024-09-17T18:31:19.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:19.660 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:19.661 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:31:19.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:31:19.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:19.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:20.104 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:21.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:21.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:21.105 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:31:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:31:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:21.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:21.530 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:22.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:22.531 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:22.531 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:31:22.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:31:22.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:22.532 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:22.917 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:23.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:23.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:23.919 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:31:23.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:31:23.919 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:23.920 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:24.206 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:24.204+0000 7fde94e1e640 -1 osd.0 74 log_to_monitors true 2024-09-17T18:31:24.365 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:25.366 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:25.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:25.367 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:31:25.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:31:25.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:25.367 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:25.789 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 85 up_thru 85 down_at 75 last_clean_interval [61,74) [v2:127.0.0.1:6802/2807326878,v1:127.0.0.1:6803/2807326878] [v2:127.0.0.1:6804/2807326878,v1:127.0.0.1:6805/2807326878] exists,up 9900589b-54ca-44ac-82b2-67b73c78870b 2024-09-17T18:31:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:31:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:31:25.789 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:31:25.790 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=145987 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:31:25.793 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:31:25.794 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:31:25.795 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.1 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:31:25.796 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:31:25.818 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:25.816+0000 7f6347818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:25.820 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:25.817+0000 7f6347818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:25.821 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:25.819+0000 7f6347818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:26.216 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:31:26.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:31:26.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:26.217 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:31:26.641 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:27.182 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:27.180+0000 7f6347818640 -1 Falling back to public interface 2024-09-17T18:31:27.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:27.642 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:27.643 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:31:27.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:31:27.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:27.643 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:31:28.064 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:29.065 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:29.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:29.066 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:31:29.066 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:31:29.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:29.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:31:29.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:30.413 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:30.411+0000 7f6347818640 -1 osd.1 77 log_to_monitors true 2024-09-17T18:31:30.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:30.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:30.497 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:31:30.497 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:31:30.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:30.498 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:31:30.996 INFO:tasks.workunit.client.0.smithi157.stdout:osd.1 up in weight 1 up_from 88 up_thru 75 down_at 78 last_clean_interval [64,77) [v2:127.0.0.1:6810/1017383796,v1:127.0.0.1:6811/1017383796] [v2:127.0.0.1:6812/1017383796,v1:127.0.0.1:6813/1017383796] exists,up 98a75569-9462-4683-ac9e-88daa1cecab8 2024-09-17T18:31:30.996 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:31:30.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:31:30.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:31:30.997 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=146990 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:31:31.000 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:31:31.001 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:31:31.002 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:31:31.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-17T18:31:31.003 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:31:31.003 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.2 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:31:31.004 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:31:31.025 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:31.023+0000 7fd6cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:31.111 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:31.109+0000 7fd6cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:31.112 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:31.110+0000 7fd6cb811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:31.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:31:31.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:31:31.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:31:31.469 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:31.470 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:31:31.926 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:32.211 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:32.209+0000 7fd6cb811640 -1 Falling back to public interface 2024-09-17T18:31:32.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:32.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:32.928 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:31:32.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:31:32.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:32.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:31:33.344 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:34.345 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:34.346 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:31:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:31:34.346 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:34.347 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:31:34.366 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:34.364+0000 7fd6cb811640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:34.645 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:34.642+0000 7fd6cb811640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:34.803 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:35.383 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:35.381+0000 7fd6cb811640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:35.804 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:35.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:35.805 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:31:35.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:31:35.805 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:35.806 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:31:35.900 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:35.897+0000 7fd6cb811640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:36.204 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:36.253 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:36.250+0000 7fd6cb811640 -1 osd.2 80 log_to_monitors true 2024-09-17T18:31:37.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:37.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:37.205 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:31:37.205 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:31:37.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:37.206 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:31:37.621 INFO:tasks.workunit.client.0.smithi157.stdout:osd.2 up in weight 1 up_from 91 up_thru 91 down_at 81 last_clean_interval [68,80) [v2:127.0.0.1:6818/879055741,v1:127.0.0.1:6819/879055741] [v2:127.0.0.1:6820/879055741,v1:127.0.0.1:6821/879055741] exists,up a54e7c67-1c45-44f6-85c2-ab26ae7d9a51 2024-09-17T18:31:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:31:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:31:37.621 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:31:37.622 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=147951 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:31:37.625 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:31:37.626 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:31:37.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-17T18:31:37.628 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.3 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:31:37.629 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:31:37.651 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:37.648+0000 7fcd24125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:37.653 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:37.650+0000 7fcd24125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:37.654 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:37.652+0000 7fcd24125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:38.108 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:31:38.109 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:31:38.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:38.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:31:38.525 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:39.250 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:39.248+0000 7fcd24125640 -1 Falling back to public interface 2024-09-17T18:31:39.526 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:39.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:39.527 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:31:39.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:31:39.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:39.527 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:31:39.925 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:40.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:40.927 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:40.927 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:31:40.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:31:40.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:40.928 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:31:41.162 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:41.160+0000 7fcd24125640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:41.354 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:41.675 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:41.672+0000 7fcd24125640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:42.355 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:42.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:42.356 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:31:42.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:31:42.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:42.356 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:31:42.461 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:42.459+0000 7fcd24125640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:42.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:42.977 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:42.974+0000 7fcd24125640 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-17T18:31:43.234 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:31:43.232+0000 7fcd24125640 -1 osd.3 82 log_to_monitors true 2024-09-17T18:31:43.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:43.822 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:43.822 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:31:43.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:31:43.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:43.823 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:31:44.409 INFO:tasks.workunit.client.0.smithi157.stdout:osd.3 up in weight 1 up_from 96 up_thru 96 down_at 83 last_clean_interval [72,82) [v2:127.0.0.1:6826/3888655018,v1:127.0.0.1:6827/3888655018] [v2:127.0.0.1:6828/3888655018,v1:127.0.0.1:6829/3888655018] exists,up 23e5d369-031d-4c43-9a77-1e2a6bf5ec55 2024-09-17T18:31:44.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:31:44.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:31:44.409 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:31:44.410 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-17T18:31:44.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=148907 2024-09-17T18:31:44.410 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-17T18:31:44.441 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:31:44.441 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-17T18:31:44.441 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_149865 2024-09-17T18:31:44.441 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:31:44.441 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:31:45.441 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 90 74 0.289032 0.289062 0.255219 0.201922 2024-09-17T18:31:46.441 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 186 170 0.332001 0.375 0.206508 0.17852 2024-09-17T18:31:47.441 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 278 262 0.341115 0.359375 0.141256 0.180884 2024-09-17T18:31:48.441 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 379 363 0.354455 0.394531 0.138893 0.17222 2024-09-17T18:31:49.441 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 504 488 0.38121 0.488281 0.102115 0.16155 2024-09-17T18:31:50.442 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 610 594 0.386676 0.414062 0.17506 0.159238 2024-09-17T18:31:51.442 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 713 697 0.388909 0.402344 0.141032 0.15976 2024-09-17T18:31:52.442 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 802 786 0.383747 0.347656 0.191977 0.161294 2024-09-17T18:31:53.442 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 931 915 0.39709 0.503906 0.117768 0.156813 2024-09-17T18:31:54.442 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 1045 1029 0.401908 0.445312 0.117634 0.154466 2024-09-17T18:31:55.442 INFO:tasks.workunit.client.0.smithi157.stdout: 11 16 1150 1134 0.402656 0.410156 0.141718 0.154269 2024-09-17T18:31:56.442 INFO:tasks.workunit.client.0.smithi157.stdout: 12 16 1253 1237 0.402627 0.402344 0.110512 0.154558 2024-09-17T18:31:57.442 INFO:tasks.workunit.client.0.smithi157.stdout: 13 16 1361 1345 0.404104 0.421875 0.156047 0.151934 2024-09-17T18:31:58.442 INFO:tasks.workunit.client.0.smithi157.stdout: 14 16 1457 1441 0.402022 0.375 0.134019 0.15502 2024-09-17T18:31:59.443 INFO:tasks.workunit.client.0.smithi157.stdout: 15 16 1569 1553 0.404382 0.4375 0.117285 0.153998 2024-09-17T18:32:00.443 INFO:tasks.workunit.client.0.smithi157.stdout: 16 16 1676 1660 0.405229 0.417969 0.136003 0.153392 2024-09-17T18:32:01.443 INFO:tasks.workunit.client.0.smithi157.stdout: 17 16 1803 1787 0.410571 0.496094 0.116778 0.151716 2024-09-17T18:32:02.443 INFO:tasks.workunit.client.0.smithi157.stdout: 18 16 1904 1888 0.409677 0.394531 0.116794 0.151805 2024-09-17T18:32:03.443 INFO:tasks.workunit.client.0.smithi157.stdout: 19 16 1986 1970 0.404971 0.320312 0.117112 0.15353 2024-09-17T18:32:04.443 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T18:32:04.442539+0000 min lat: 0.0711432 max lat: 0.620618 avg lat: 0.153712 2024-09-17T18:32:04.444 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:32:04.444 INFO:tasks.workunit.client.0.smithi157.stdout: 20 16 2091 2075 0.405227 0.410156 0.133782 0.153712 2024-09-17T18:32:05.443 INFO:tasks.workunit.client.0.smithi157.stdout: 21 16 2204 2188 0.406947 0.441406 0.133038 0.153195 2024-09-17T18:32:06.444 INFO:tasks.workunit.client.0.smithi157.stdout: 22 16 2310 2294 0.407268 0.414062 0.133699 0.153096 2024-09-17T18:32:07.444 INFO:tasks.workunit.client.0.smithi157.stdout: 23 16 2432 2416 0.410279 0.476562 0.117593 0.151951 2024-09-17T18:32:08.444 INFO:tasks.workunit.client.0.smithi157.stdout: 24 16 2538 2522 0.410434 0.414062 0.107068 0.151868 2024-09-17T18:32:09.444 INFO:tasks.workunit.client.0.smithi157.stdout: 25 16 2626 2610 0.407765 0.34375 0.0832149 0.153078 2024-09-17T18:32:10.444 INFO:tasks.workunit.client.0.smithi157.stdout: 26 16 2741 2725 0.409358 0.449219 0.10187 0.152316 2024-09-17T18:32:11.444 INFO:tasks.workunit.client.0.smithi157.stdout: 27 16 2860 2844 0.411412 0.464844 0.184568 0.151559 2024-09-17T18:32:12.444 INFO:tasks.workunit.client.0.smithi157.stdout: 28 16 2972 2956 0.412342 0.4375 0.117402 0.151225 2024-09-17T18:32:13.444 INFO:tasks.workunit.client.0.smithi157.stdout: 29 16 3091 3075 0.414151 0.464844 0.0997932 0.150564 2024-09-17T18:32:14.444 INFO:tasks.workunit.client.0.smithi157.stdout: 30 16 3212 3196 0.4161 0.472656 0.133491 0.149962 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 30.118 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 3212 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.41659 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0517586 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.503906 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.289062 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 106 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 13.2502 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 129 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 74 2024-09-17T18:32:15.445 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.149962 2024-09-17T18:32:15.446 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0500043 2024-09-17T18:32:15.446 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.620618 2024-09-17T18:32:15.446 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.0711432 2024-09-17T18:32:15.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-17T18:32:15.494 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:32:15.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:32:15.495 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:32:15.495 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:32:15.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:32:15.496 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:32:15.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:32:15.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:32:15.496 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:32:15.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:32:15.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:32:15.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:32:15.592 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:32:15.593 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:32:15.593 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:32:15.980 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr:1 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr:2 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr:3' 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:15.981 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:32:16.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220173 2024-09-17T18:32:16.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220173 2024-09-17T18:32:16.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-365072220173' 2024-09-17T18:32:16.095 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:16.096 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:32:16.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122060 2024-09-17T18:32:16.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122060 2024-09-17T18:32:16.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-365072220173 1-377957122060' 2024-09-17T18:32:16.207 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:16.207 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:32:16.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023946 2024-09-17T18:32:16.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023946 2024-09-17T18:32:16.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-365072220173 1-377957122060 2-390842023946' 2024-09-17T18:32:16.320 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:16.320 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:32:16.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860425 2024-09-17T18:32:16.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860425 2024-09-17T18:32:16.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-365072220173 1-377957122060 2-390842023946 3-412316860425' 2024-09-17T18:32:16.432 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:16.432 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-365072220173 2024-09-17T18:32:16.433 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:16.433 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:32:16.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-365072220173 2024-09-17T18:32:16.434 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:16.435 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220173 2024-09-17T18:32:16.435 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 365072220173 2024-09-17T18:32:16.436 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 365072220173' 2024-09-17T18:32:16.436 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:32:16.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220173 -lt 365072220173 2024-09-17T18:32:16.830 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:16.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-377957122060 2024-09-17T18:32:16.831 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:16.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:32:16.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-377957122060 2024-09-17T18:32:16.832 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:16.833 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122060 2024-09-17T18:32:16.833 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.1 seq 377957122060 2024-09-17T18:32:16.834 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 377957122060' 2024-09-17T18:32:16.834 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:32:17.258 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122060 -lt 377957122060 2024-09-17T18:32:17.259 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:17.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-390842023946 2024-09-17T18:32:17.259 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:17.260 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:32:17.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-390842023946 2024-09-17T18:32:17.261 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:17.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023946 2024-09-17T18:32:17.262 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 390842023946' 2024-09-17T18:32:17.262 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.2 seq 390842023946 2024-09-17T18:32:17.262 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:32:17.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023947 -lt 390842023946 2024-09-17T18:32:17.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:17.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-412316860425 2024-09-17T18:32:17.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:17.686 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:32:17.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-412316860425 2024-09-17T18:32:17.687 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:17.688 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.3 seq 412316860425 2024-09-17T18:32:17.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860425 2024-09-17T18:32:17.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 412316860425' 2024-09-17T18:32:17.688 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:32:18.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860425 -lt 412316860425 2024-09-17T18:32:18.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:32:18.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:18.112 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:18.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T18:32:18.628 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:32:18.628 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:32:18.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:32:18.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:32:18.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:32:18.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:32:18.629 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:32:19.034 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T18:32:19.034 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:32:19.035 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:19.035 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:19.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T18:32:19.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:32:19.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:32:19.559 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:19.560 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:19.668 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:19.668 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:32:19.669 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:32:19.669 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:32:19.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:32:19.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:32:19.671 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:32:19.671 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:19.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:32:19.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:32:19.672 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:19.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:32:19.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:32:19.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:32:20.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:32:20.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.103 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.103 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:20.105 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:20.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:20.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:20.106 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:20.106 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:32:20.107 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:32:20.107 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:32:20.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:32:20.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:32:20.109 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:32:20.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:20.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:32:20.110 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:32:20.110 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:20.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:32:20.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:32:20.112 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:32:20.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:32:20.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.113 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:32:20.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:32:20.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:32:20.114 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-17T18:32:20.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:32:20.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.115 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.116 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102844 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-17T18:32:20.118 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-17T18:32:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none ' 2024-09-17T18:32:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-17T18:32:20.120 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-17T18:32:20.121 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:32:20.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:32:20.769 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:32:20.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:32:20.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:32:20.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:32:20.770 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:32:20.771 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:32:20.771 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:32:20.772 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get fsid 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:32:20.854 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:20.855 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:20.855 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:32:20.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:32:20.855 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get mon_host 2024-09-17T18:32:20.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-17T18:32:20.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:20.933 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:32:20.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:32:20.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:32:20.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-17T18:32:20.934 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:32:21.783 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:32:21.783 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:21.783 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:21.783 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:21.784 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:21.784 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:21.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:32:21.784 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:32:21.784 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-17T18:32:21.818 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:21.819 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:32:21.820 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6b04f3e7-d511-40e1-b944-bcf9091818ff 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6b04f3e7-d511-40e1-b944-bcf9091818ff' 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 6b04f3e7-d511-40e1-b944-bcf9091818ff 2024-09-17T18:32:21.821 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:32:21.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1y+lm2Ar5MRAAmv8AOP5zUP0rLZS0U+vnqA== 2024-09-17T18:32:21.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1y+lm2Ar5MRAAmv8AOP5zUP0rLZS0U+vnqA=="}' 2024-09-17T18:32:21.840 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6b04f3e7-d511-40e1-b944-bcf9091818ff -i td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:32:22.102 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:32:22.111 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:32:22.113 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1y+lm2Ar5MRAAmv8AOP5zUP0rLZS0U+vnqA== --osd-uuid 6b04f3e7-d511-40e1-b944-bcf9091818ff 2024-09-17T18:32:22.146 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:22.144+0000 7f044c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:22.171 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:22.168+0000 7f044c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:22.172 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:22.169+0000 7f044c80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:22.173 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:22.170+0000 7f044c80d640 -1 bdev(0x55ff2a346800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:32:22.173 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:22.170+0000 7f044c80d640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-17T18:32:29.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-17T18:32:29.445 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:32:29.446 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:32:29.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:32:29.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:32:29.991 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:32:29.991 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:32:29.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:32:29.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:32:29.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:32:29.992 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:32:30.023 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:30.020+0000 7f2de75fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:30.033 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:30.031+0000 7f2de75fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:30.034 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:30.032+0000 7f2de75fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:30.447 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:30.448 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:32:30.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:32:30.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:30.449 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:30.873 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:31.381 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:31.379+0000 7f2de75fd640 -1 Falling back to public interface 2024-09-17T18:32:31.874 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:31.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:31.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:32:31.875 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:32:31.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:31.875 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:32.299 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:33.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:33.300 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:33.301 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:32:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:32:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:33.301 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:33.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:34.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:34.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:34.735 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:32:34.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:32:34.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:34.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:35.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:35.386 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:35.383+0000 7f2de75fd640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:32:36.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:36.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:36.138 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:32:36.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:32:36.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:36.139 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:37.234 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:38.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:38.236 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:38.236 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:32:38.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:32:38.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:38.237 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:39.168 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:39.390 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:32:39.389+0000 7f2de75fd640 -1 bdev(0x55f079fed800 td/osd-bluefs-volume-ops/0/block) aio_submit retries 8 2024-09-17T18:32:40.169 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:40.170 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:32:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:32:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:40.170 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:40.567 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/662299421,v1:127.0.0.1:6803/662299421] [v2:127.0.0.1:6804/662299421,v1:127.0.0.1:6805/662299421] exists,up 6b04f3e7-d511-40e1-b944-bcf9091818ff 2024-09-17T18:32:40.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:32:40.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:32:40.568 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:32:40.568 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:32:40.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=151740 2024-09-17T18:32:40.569 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-17T18:32:45.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-17T18:32:45.570 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-17T18:32:46.114 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-17T18:32:46.126 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:32:47.127 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-17T18:32:47.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-17T18:32:47.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-17T18:32:47.128 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-17T18:32:47.167 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:32:47.168 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-17T18:32:47.169 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_153030 2024-09-17T18:32:47.198 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:32:47.198 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:32:48.198 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 17 1 3.99961 4 0.239142 0.239142 2024-09-17T18:32:49.198 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 49 33 65.9922 128 0.76825 0.868188 2024-09-17T18:32:50.199 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 65 49 65.3249 64 0.82668 0.844686 2024-09-17T18:32:51.199 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 80 64 63.992 60 1.34948 0.965436 2024-09-17T18:32:52.199 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 81 65 51.9937 4 1.47016 0.973201 2024-09-17T18:32:53.199 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 113 97 64.6589 128 0.793173 0.95396 2024-09-17T18:32:54.199 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 129 113 64.5636 64 1.05277 0.970385 2024-09-17T18:32:55.199 INFO:tasks.workunit.client.0.smithi157.stdout: 8 16 144 128 63.9921 60 1.11535 0.987363 2024-09-17T18:32:56.199 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 145 129 57.3263 4 1.12819 0.988454 2024-09-17T18:32:57.200 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 177 161 64.392 128 0.684041 0.983257 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 10.9639 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 177 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4194304 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4194304 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 64.5757 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 50.6825 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 128 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 4 2024-09-17T18:32:58.200 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 16 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 12.6706 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 32 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 1 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.977623 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.215645 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 1.47016 2024-09-17T18:32:58.201 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.239142 2024-09-17T18:32:58.243 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-17T18:33:08.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-17T18:33:08.246 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-17T18:33:08.381 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1062207488 2024-09-17T18:33:08.382 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-17T18:33:08.382 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-17T18:33:08.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=184418304 2024-09-17T18:33:08.506 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-17T18:33:08.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 184418304 -eq 0 2024-09-17T18:33:08.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-17T18:33:08.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 184418304 -gt 0 2024-09-17T18:33:08.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151740 2024-09-17T18:33:08.507 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-17T18:33:09.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151740 2024-09-17T18:33:09.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (151740) - No such process 2024-09-17T18:33:09.508 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-17T18:33:09.942 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:33:09.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-17T18:33:09.977 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:33:20.032 INFO:tasks.workunit.client.0.smithi157.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-17T18:33:20.047 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-17T18:33:20.074 INFO:tasks.workunit.client.0.smithi157.stdout:inferring bluefs devices from bluestore path 2024-09-17T18:33:22.100 INFO:tasks.workunit.client.0.smithi157.stdout:1 : device size 0xffffe000 : using 0x4a6f8000(1.2 GiB) 2024-09-17T18:33:22.109 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-17T18:33:24.672 INFO:tasks.workunit.client.0.smithi157.stdout:fsck success 2024-09-17T18:33:24.711 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:33:24.712 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:33:24.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:33:24.714 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:33:24.715 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:33:24.743 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:24.740+0000 7f62da2d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:24.745 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:24.742+0000 7f62da2d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:24.746 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:24.743+0000 7f62da2d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:25.136 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:25.137 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:33:25.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:33:25.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:25.138 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:25.562 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:26.097 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:26.093+0000 7f62da2d4640 -1 Falling back to public interface 2024-09-17T18:33:26.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:26.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:26.563 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:33:26.563 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:33:26.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:26.564 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:26.997 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:27.998 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:27.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:27.999 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:33:27.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:33:27.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:27.999 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:28.441 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:29.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:29.442 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:29.443 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:33:29.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:33:29.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:29.443 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:29.893 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:30.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:30.895 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:30.895 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:33:30.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:33:30.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:30.896 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:31.295 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:32.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:32.296 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:32.296 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:33:32.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:33:32.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:32.297 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:32.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:33.550 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:33.547+0000 7f62da2d4640 -1 osd.0 10 log_to_monitors true 2024-09-17T18:33:33.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:33.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:33.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:33:33.689 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:33:33.689 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:33.690 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:34.848 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:35.849 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:35.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:35.850 INFO:tasks.workunit.client.0.smithi157.stdout:7 2024-09-17T18:33:35.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:33:35.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:35.850 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:37.067 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:38.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:38.069 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:38.069 INFO:tasks.workunit.client.0.smithi157.stdout:8 2024-09-17T18:33:38.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:33:38.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:38.070 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:38.513 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/1147263174,v1:127.0.0.1:6803/1147263174] [v2:127.0.0.1:6804/1147263174,v1:127.0.0.1:6805/1147263174] exists,up 6b04f3e7-d511-40e1-b944-bcf9091818ff 2024-09-17T18:33:38.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:33:38.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:33:38.514 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:33:38.514 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:33:38.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=154261 2024-09-17T18:33:38.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-17T18:33:38.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:33:38.515 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:33:38.516 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:33:38.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:33:38.516 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:33:38.516 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:33:38.516 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:33:38.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:33:38.517 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:33:38.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:33:38.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:33:38.603 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:33:38.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:33:38.604 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:33:38.604 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:33:39.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-17T18:33:39.023 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:33:39.024 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:33:39.024 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:33:39.141 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574852 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574852 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-55834574852' 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-55834574852 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:33:39.142 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:33:39.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-55834574852 2024-09-17T18:33:39.143 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:33:39.144 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574852 2024-09-17T18:33:39.145 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-17T18:33:39.145 INFO:tasks.workunit.client.0.smithi157.stdout:waiting osd.0 seq 55834574852 2024-09-17T18:33:39.145 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:33:39.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 55834574852 2024-09-17T18:33:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:33:40.584 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:33:40.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:33:41.017 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-17T18:33:41.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:33:41.017 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:33:41.018 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:33:41.551 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:33:41.954 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-17T18:33:41.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:33:41.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:33:41.955 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:33:42.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:33:42.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:33:42.478 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:33:42.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:33:42.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:33:42.478 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:33:42.582 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:33:42.583 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:33:42.583 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:33:42.584 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:33:42.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:33:42.585 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:33:42.585 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:33:42.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:33:42.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:33:42.586 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:33:42.586 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:33:42.587 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:33:42.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:33:42.588 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:33:42.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:33:42.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:42.670 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:42.670 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:42.672 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:33:42.673 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:33:42.674 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:33:42.675 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:33:42.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:33:42.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:33:42.676 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:33:42.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:33:42.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:33:42.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:33:42.678 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:33:42.678 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:33:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:33:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:33:42.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:33:42.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:42.680 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:42.680 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:33:42.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:33:42.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:33:42.681 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-17T18:33:42.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:33:42.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:42.682 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:42.683 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.102844 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-17T18:33:42.685 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-17T18:33:42.686 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-17T18:33:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=0d5d21c1-a932-468e-9549-18a6bae9afb3 --auth-supported=none ' 2024-09-17T18:33:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-17T18:33:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-17T18:33:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-17T18:33:42.687 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-17T18:33:42.688 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:43.124 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:43.125 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:33:43.316 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:33:43.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:33:43.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:33:43.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:33:43.318 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:33:43.319 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get fsid 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:33:43.397 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:33:43.398 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:43.398 INFO:tasks.workunit.client.0.smithi157.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:43.398 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.102844/ceph-mon.a.asok 2024-09-17T18:33:43.398 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:33:43.398 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.102844/ceph-mon.a.asok config get mon_host 2024-09-17T18:33:43.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-17T18:33:43.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:43.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:33:43.476 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:33:43.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:33:43.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-17T18:33:43.477 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:33:44.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:33:44.337 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:33:44.338 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:33:44.376 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=0d5d21c1-a932-468e-9549-18a6bae9afb3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-17T18:33:44.377 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:33:44.378 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-17T18:33:44.379 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:33:44.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=34cd34b5-37f1-4b22-9fd0-24aedad89c1d 2024-09-17T18:33:44.380 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 34cd34b5-37f1-4b22-9fd0-24aedad89c1d' 2024-09-17T18:33:44.380 INFO:tasks.workunit.client.0.smithi157.stdout:add osd0 34cd34b5-37f1-4b22-9fd0-24aedad89c1d 2024-09-17T18:33:44.380 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:33:44.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCIy+lmdM30FxAAr9IuBXj+Go8u/AJQI9Lr6Q== 2024-09-17T18:33:44.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCIy+lmdM30FxAAr9IuBXj+Go8u/AJQI9Lr6Q=="}' 2024-09-17T18:33:44.404 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 34cd34b5-37f1-4b22-9fd0-24aedad89c1d -i td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:33:44.666 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:33:44.676 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-17T18:33:44.677 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=0d5d21c1-a932-468e-9549-18a6bae9afb3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIy+lmdM30FxAAr9IuBXj+Go8u/AJQI9Lr6Q== --osd-uuid 34cd34b5-37f1-4b22-9fd0-24aedad89c1d 2024-09-17T18:33:44.717 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:44.715+0000 7fadd15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:44.752 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:44.750+0000 7fadd15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:44.753 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:44.752+0000 7fadd15cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:44.754 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:44.752+0000 7fadd15cc640 -1 bdev(0x55ea67a98800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:33:44.754 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:44.752+0000 7fadd15cc640 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-17T18:33:52.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-17T18:33:52.280 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:33:52.281 INFO:tasks.workunit.client.0.smithi157.stdout:adding osd0 key to auth repository 2024-09-17T18:33:52.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:33:52.281 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:33:52.830 INFO:tasks.workunit.client.0.smithi157.stdout:start osd.0 2024-09-17T18:33:52.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:33:52.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1ee1cbc9-ad95-402d-8838-f941ccf6a65d --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=c53d62cf-6c82-46db-8231-7098e9afec95 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=0d5d21c1-a932-468e-9549-18a6bae9afb3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102844/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:33:52.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:33:52.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:33:52.831 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:33:52.868 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:52.867+0000 7f6233e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:52.870 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:52.869+0000 7f6233e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:52.871 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:52.870+0000 7f6233e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:33:53.277 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:53.278 INFO:tasks.workunit.client.0.smithi157.stdout:0 2024-09-17T18:33:53.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:33:53.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:53.278 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:53.702 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:54.215 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:54.214+0000 7f6233e0b640 -1 Falling back to public interface 2024-09-17T18:33:54.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:54.704 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:54.704 INFO:tasks.workunit.client.0.smithi157.stdout:1 2024-09-17T18:33:54.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:33:54.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:54.705 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:55.130 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:56.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:56.131 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:56.132 INFO:tasks.workunit.client.0.smithi157.stdout:2 2024-09-17T18:33:56.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:33:56.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:56.132 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:56.655 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:57.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:57.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:57.657 INFO:tasks.workunit.client.0.smithi157.stdout:3 2024-09-17T18:33:57.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:33:57.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:57.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:33:58.054 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:58.639 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:33:58.638+0000 7f6233e0b640 -1 osd.0 0 log_to_monitors true 2024-09-17T18:33:59.055 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:59.056 INFO:tasks.workunit.client.0.smithi157.stdout:4 2024-09-17T18:33:59.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:59.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:33:59.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:59.056 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:00.656 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:01.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:01.657 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:01.657 INFO:tasks.workunit.client.0.smithi157.stdout:5 2024-09-17T18:34:01.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:34:01.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:01.658 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:02.661 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:03.662 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:03.663 INFO:tasks.workunit.client.0.smithi157.stdout:6 2024-09-17T18:34:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:34:03.663 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:03.664 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:04.097 INFO:tasks.workunit.client.0.smithi157.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3326463124,v1:127.0.0.1:6803/3326463124] [v2:127.0.0.1:6804/3326463124,v1:127.0.0.1:6805/3326463124] exists,up 34cd34b5-37f1-4b22-9fd0-24aedad89c1d 2024-09-17T18:34:04.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:34:04.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:34:04.098 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:34:04.098 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-17T18:34:04.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=157020 2024-09-17T18:34:04.099 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-17T18:34:09.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-17T18:34:09.100 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-17T18:34:09.603 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'foo' already exists 2024-09-17T18:34:09.614 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:34:10.615 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-17T18:34:10.659 INFO:tasks.workunit.client.0.smithi157.stdout:hints = 1 2024-09-17T18:34:10.659 INFO:tasks.workunit.client.0.smithi157.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-17T18:34:10.659 INFO:tasks.workunit.client.0.smithi157.stdout:Object prefix: benchmark_data_smithi157_158056 2024-09-17T18:34:10.659 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:34:10.659 INFO:tasks.workunit.client.0.smithi157.stdout: 0 0 0 0 0 0 - 0 2024-09-17T18:34:11.659 INFO:tasks.workunit.client.0.smithi157.stdout: 1 16 145 129 0.503861 0.503906 0.117152 0.114553 2024-09-17T18:34:12.660 INFO:tasks.workunit.client.0.smithi157.stdout: 2 16 273 257 0.501905 0.5 0.167432 0.115972 2024-09-17T18:34:13.660 INFO:tasks.workunit.client.0.smithi157.stdout: 3 16 464 448 0.583278 0.746094 0.109155 0.106929 2024-09-17T18:34:14.660 INFO:tasks.workunit.client.0.smithi157.stdout: 4 16 592 576 0.562439 0.5 0.176097 0.110547 2024-09-17T18:34:15.660 INFO:tasks.workunit.client.0.smithi157.stdout: 5 16 752 736 0.574938 0.625 0.0924728 0.108558 2024-09-17T18:34:16.660 INFO:tasks.workunit.client.0.smithi157.stdout: 6 16 880 864 0.562439 0.5 0.151034 0.110601 2024-09-17T18:34:17.660 INFO:tasks.workunit.client.0.smithi157.stdout: 7 16 1008 992 0.553511 0.5 0.100657 0.112784 2024-09-17T18:34:18.660 INFO:tasks.workunit.client.0.smithi157.stdout: 8 15 1124 1109 0.541447 0.457031 0.19264 0.113435 2024-09-17T18:34:19.660 INFO:tasks.workunit.client.0.smithi157.stdout: 9 16 1281 1265 0.548986 0.609375 0.0920232 0.112762 2024-09-17T18:34:20.660 INFO:tasks.workunit.client.0.smithi157.stdout: 10 16 1425 1409 0.550332 0.5625 0.184214 0.111367 2024-09-17T18:34:21.661 INFO:tasks.workunit.client.0.smithi157.stdout: 11 16 1584 1568 0.556757 0.621094 0.0924457 0.11217 2024-09-17T18:34:22.661 INFO:tasks.workunit.client.0.smithi157.stdout: 12 16 1712 1696 0.552023 0.5 0.167712 0.112364 2024-09-17T18:34:23.661 INFO:tasks.workunit.client.0.smithi157.stdout: 13 16 1872 1856 0.557632 0.625 0.0838742 0.111935 2024-09-17T18:34:24.661 INFO:tasks.workunit.client.0.smithi157.stdout: 14 16 2001 1985 0.553791 0.503906 0.200964 0.111616 2024-09-17T18:34:25.661 INFO:tasks.workunit.client.0.smithi157.stdout: 15 16 2161 2145 0.558531 0.625 0.092106 0.111314 2024-09-17T18:34:26.661 INFO:tasks.workunit.client.0.smithi157.stdout: 16 16 2289 2273 0.554869 0.5 0.175825 0.11195 2024-09-17T18:34:27.661 INFO:tasks.workunit.client.0.smithi157.stdout: 17 16 2448 2432 0.55876 0.621094 0.0925394 0.111514 2024-09-17T18:34:28.661 INFO:tasks.workunit.client.0.smithi157.stdout: 18 16 2576 2560 0.555493 0.5 0.183514 0.112414 2024-09-17T18:34:29.661 INFO:tasks.workunit.client.0.smithi157.stdout: 19 16 2736 2720 0.559147 0.625 0.100539 0.111519 2024-09-17T18:34:30.661 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-17T18:34:30.660844+0000 min lat: 0.0742995 max lat: 0.359454 avg lat: 0.11204 2024-09-17T18:34:30.662 INFO:tasks.workunit.client.0.smithi157.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-17T18:34:30.662 INFO:tasks.workunit.client.0.smithi157.stdout: 20 16 2864 2848 0.556187 0.5 0.201067 0.11204 2024-09-17T18:34:31.662 INFO:tasks.workunit.client.0.smithi157.stdout: 21 16 3009 2993 0.556669 0.566406 0.0837122 0.11189 2024-09-17T18:34:32.662 INFO:tasks.workunit.client.0.smithi157.stdout: 22 16 3168 3152 0.559594 0.621094 0.183786 0.111602 2024-09-17T18:34:33.662 INFO:tasks.workunit.client.0.smithi157.stdout: 23 16 3313 3297 0.559886 0.566406 0.117224 0.111327 2024-09-17T18:34:34.662 INFO:tasks.workunit.client.0.smithi157.stdout: 24 16 3440 3424 0.557226 0.496094 0.17676 0.111846 2024-09-17T18:34:35.662 INFO:tasks.workunit.client.0.smithi157.stdout: 25 16 3585 3569 0.557591 0.566406 0.117274 0.111697 2024-09-17T18:34:36.662 INFO:tasks.workunit.client.0.smithi157.stdout: 26 16 3729 3713 0.557778 0.5625 0.124522 0.111867 2024-09-17T18:34:37.662 INFO:tasks.workunit.client.0.smithi157.stdout: 27 16 3905 3889 0.56258 0.6875 0.0918986 0.110865 2024-09-17T18:34:38.662 INFO:tasks.workunit.client.0.smithi157.stdout: 28 16 4048 4032 0.562434 0.558594 0.167592 0.110924 2024-09-17T18:34:39.663 INFO:tasks.workunit.client.0.smithi157.stdout: 29 16 4208 4192 0.56459 0.625 0.108657 0.110623 2024-09-17T18:34:40.663 INFO:tasks.workunit.client.0.smithi157.stdout: 30 16 4337 4321 0.562564 0.503906 0.201092 0.11055 2024-09-17T18:34:41.663 INFO:tasks.workunit.client.0.smithi157.stdout:Total time run: 30.0756 2024-09-17T18:34:41.663 INFO:tasks.workunit.client.0.smithi157.stdout:Total writes made: 4337 2024-09-17T18:34:41.663 INFO:tasks.workunit.client.0.smithi157.stdout:Write size: 4096 2024-09-17T18:34:41.663 INFO:tasks.workunit.client.0.smithi157.stdout:Object size: 4096 2024-09-17T18:34:41.663 INFO:tasks.workunit.client.0.smithi157.stdout:Bandwidth (MB/sec): 0.563294 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Bandwidth: 0.0691085 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Max bandwidth (MB/sec): 0.746094 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Min bandwidth (MB/sec): 0.457031 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Average IOPS: 144 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev IOPS: 17.6918 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Max IOPS: 191 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Min IOPS: 117 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Average Latency(s): 0.110779 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Stddev Latency(s): 0.0312179 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Max latency(s): 0.359454 2024-09-17T18:34:41.664 INFO:tasks.workunit.client.0.smithi157.stdout:Min latency(s): 0.0742995 2024-09-17T18:34:41.713 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-17T18:34:46.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-17T18:34:46.715 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-17T18:34:46.856 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-17T18:34:46.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-17T18:34:46.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-17T18:34:46.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-17T18:34:46.857 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-17T18:34:46.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-17T18:34:46.858 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-17T18:34:46.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4277096448 2024-09-17T18:34:46.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 157020 2024-09-17T18:34:46.982 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-17T18:34:47.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 157020 2024-09-17T18:34:47.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (157020) - No such process 2024-09-17T18:34:47.983 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-17T18:34:48.530 INFO:tasks.workunit.client.0.smithi157.stderr:osd.0 is already down. 2024-09-17T18:34:48.542 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-17T18:34:48.571 INFO:tasks.workunit.client.0.smithi157.stdout:allocmap bluestore.allocmap 2024-09-17T18:34:51.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:34:51.177+0000 7f06305dc980 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-17T18:34:51.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:34:51.177+0000 7f06305dc980 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-17T18:34:51.179 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-17T18:34:51.177+0000 7f06305dc980 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-17T18:34:51.620 INFO:tasks.workunit.client.0.smithi157.stderr:allocmap failed: (1) Operation not permitted 2024-09-17T18:34:51.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-17T18:34:51.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-17T18:34:51.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-17T18:34:51.626 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bluefs-volume-ops 1 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:34:51.627 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:34:51.733 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:34:51.733 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:34:51.734 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:34:51.734 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:34:51.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:34:51.735 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:34:51.735 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:34:51.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:34:51.736 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:34:51.737 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:34:51.737 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:34:51.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:34:51.738 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-17T18:34:51.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-17T18:34:51.739 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-17T18:34:51.740 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-17T18:34:51.741 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-17T18:34:51.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:34:51.744 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:34:51.745 INFO:tasks.workunit.client.0.smithi157.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.102844 2024-09-17T18:34:51.745 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.102844 2024-09-17T18:34:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:34:51.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:34:51.746 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:34:51.747 INFO:tasks.workunit.client.0.smithi157.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-17T18:34:51.747 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-17T18:34:51.747 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T18:34:52.088 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-17T18:34:52.263 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=adf4052b52ee45ccbf9f5887e0166c41 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-17T18:34:52.265 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T18:34:52.274 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T18:34:52.274 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T18:34:52.324 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T18:34:52.332 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T18:34:52.333 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:34:52.359 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 -38us[ -18us] +/- 56ms 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 +410us[ +410us] +/- 73ms 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -346us[ -346us] +/- 78ms 2024-09-17T18:34:52.364 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T18:34:52.365 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T18:34:52.372 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T18:34:52.373 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T18:34:52.411 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T18:34:52.468 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T18:34:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:34:52.495 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T18:34:52.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T18:34:52.569 INFO:teuthology.task.internal:Duration was 2657.857532 seconds 2024-09-17T18:34:52.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T18:34:52.619 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T18:34:52.619 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T18:34:52.658 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T18:34:52.814 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T18:34:52.814 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-09-17T18:34:52.814 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T18:34:52.891 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T18:34:52.891 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T18:34:52.957 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T18:34:52.957 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T18:34:53.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T18:34:53.101 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T18:34:53.101 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T18:34:53.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T18:34:53.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:34:53.189 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-09-17T18:34:53.212 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:34:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:34:53.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T18:34:53.276 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T18:34:53.278 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909201/remote/smithi157 2024-09-17T18:34:53.279 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T18:34:56.639 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T18:34:56.640 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T18:34:56.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T18:34:56.711 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T18:34:56.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T18:34:56.744 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T18:34:56.745 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T18:34:56.762 INFO:teuthology.orchestra.run.smithi157.stdout: 265786 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 17 18:34 /home/ubuntu/cephtest 2024-09-17T18:34:56.762 INFO:teuthology.orchestra.run.smithi157.stdout: 405780 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 17 17:56 /home/ubuntu/cephtest/mnt.0 2024-09-17T18:34:56.762 INFO:teuthology.orchestra.run.smithi157.stdout: 405781 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 17 18:22 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T18:34:56.762 INFO:teuthology.orchestra.run.smithi157.stdout: 405824 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 17 18:22 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:34:56.762 INFO:teuthology.orchestra.run.smithi157.stdout: 420646 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 18:34 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-17T18:34:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:34:56.763 INFO:teuthology.orchestra.run.smithi157.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-17T18:34:56.763 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi157 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-17T18:34:56.768 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T18:34:56.794 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T18:34:56.846 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi157 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-17T18:34:56.849 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 2657.857532262802 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi157 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=adf4052b52ee45ccbf9f5887e0166c41 status: fail success: false 2024-09-17T18:34:56.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T18:34:56.937 INFO:teuthology.run:FAIL